|
| 1 | +# Arnaud Bouchez |
| 2 | + |
| 3 | +**mORMot entry to The One Billion Row Challenge in Object Pascal.** |
| 4 | + |
| 5 | +## mORMot 2 is Required |
| 6 | + |
| 7 | +This entry requires the **mORMot 2** package to compile. |
| 8 | + |
| 9 | +Download it from https://github.com/synopse/mORMot2 |
| 10 | + |
| 11 | +It is better to fork the current state of the *mORMot 2* repository, or get the latest release. |
| 12 | + |
| 13 | +## Licence Terms |
| 14 | + |
| 15 | +This code is licenced by its sole author (A. Bouchez) as MIT terms, to be used for pedagogical reasons. |
| 16 | + |
| 17 | +I am very happy to share decades of server-side performance coding techniques using FPC on x86_64. ;) |
| 18 | + |
| 19 | +## Presentation |
| 20 | + |
| 21 | +Here are the main ideas behind this implementation proposal: |
| 22 | + |
| 23 | +- **mORMot** makes cross-platform and cross-compiler support simple (e.g. `TMemMap`, `TDynArray.Sort`,`TTextWriter`, `SetThreadCpuAffinity`, `crc32c`, `ConsoleWrite` or command-line parsing); |
| 24 | +- Will memmap the entire 16GB file at once into memory (so won't work on 32-bit OS, but reduce syscalls); |
| 25 | +- Process file in parallel using several threads (configurable, with `-t=16` by default); |
| 26 | +- Fed each thread from 64MB chunks of input (because thread scheduling is unfair, it is inefficient to pre-divide the size of the whole input file into the number of threads); |
| 27 | +- Each thread manages its own data, so there is no lock until the thread is finished and data is consolidated; |
| 28 | +- Each station information (name and values) is packed into a record of exactly 64 bytes, with no external pointer/string, to match the CPU L1 cache size for efficiency; |
| 29 | +- Use a dedicated hash table for the name lookup, with direct crc32c SSE4.2 hash - when `TDynArrayHashed` is involved, it requires a transient name copy on the stack, which is noticeably slower (see last paragraph of this document); |
| 30 | +- Store values as 16-bit or 32-bit integers (i.e. temperature multiplied by 10); |
| 31 | +- Parse temperatures with a dedicated code (expects single decimal input values); |
| 32 | +- No memory allocation (e.g. no transient `string` or `TBytes`) nor any syscall is done during the parsing process to reduce contention and ensure the process is only CPU-bound and RAM-bound (we checked this with `strace` on Linux); |
| 33 | +- Pascal code was tuned to generate the best possible asm output on FPC x86_64 (which is our target); |
| 34 | +- Some dedicated x86_64 asm has been written to replace *mORMot* `crc32c` and `MemCmp` general-purpose functions and gain a last few percents (nice to have); |
| 35 | +- Can optionally output timing statistics and hash value on the console to debug and refine settings (with the `-v` command line switch); |
| 36 | +- Can optionally set each thread affinity to a single core (with the `-a` command line switch). |
| 37 | + |
| 38 | +The "64 bytes cache line" trick is quite unique among all implementations of the "1brc" I have seen in any language - and it does make a noticeable difference in performance. The L1 cache is well known to be the main bottleneck for any efficient in-memory process. We are very lucky the station names are just big enough to fill no more than 64 bytes, with min/max values reduced as 16-bit smallint - resulting in temperature range of -3276.7..+3276.8 which seems fair on our planet according to the IPCC. ;) |
| 39 | + |
| 40 | +## Usage |
| 41 | + |
| 42 | +If you execute the `abouchez` executable without any parameter, it will give you some hints about its usage (using *mORMot* `TCommandLine` abilities): |
| 43 | + |
| 44 | +``` |
| 45 | +ab@dev:~/dev/github/1brc-ObjectPascal/bin$ ./abouchez |
| 46 | +The mORMot One Billion Row Challenge |
| 47 | +
|
| 48 | +Usage: abouchez <filename> [options] [params] |
| 49 | +
|
| 50 | + <filename> the data source filename |
| 51 | +
|
| 52 | +Options: |
| 53 | + -v, --verbose generate verbose output with timing |
| 54 | + -a, --affinity force thread affinity to a single CPU core |
| 55 | + -h, --help display this help |
| 56 | +
|
| 57 | +Params: |
| 58 | + -t, --threads <number> (default 16) |
| 59 | + number of threads to run |
| 60 | +``` |
| 61 | +We will use these command-line switches for local (dev PC), and benchmark (challenge HW) analysis. |
| 62 | + |
| 63 | +## Local Analysis |
| 64 | + |
| 65 | +On my PC, it takes less than 5 seconds to process the 16GB file with 8/10 threads. |
| 66 | + |
| 67 | +Let's compare `abouchez` with a solid multi-threaded entry using file buffer reads and no memory map (like `sbalazs`), using the `time` command on Linux: |
| 68 | + |
| 69 | +``` |
| 70 | +ab@dev:~/dev/github/1brc-ObjectPascal/bin$ time ./abouchez measurements.txt -t=10 >resmrel5.txt |
| 71 | +
|
| 72 | +real 0m4,216s |
| 73 | +user 0m38,789s |
| 74 | +sys 0m0,632s |
| 75 | +
|
| 76 | +ab@dev:~/dev/github/1brc-ObjectPascal/bin$ time ./sbalazs measurements.txt 20 >ressb6.txt |
| 77 | +
|
| 78 | +real 0m25,330s |
| 79 | +user 6m44,853s |
| 80 | +sys 0m31,167s |
| 81 | +``` |
| 82 | +We used 20 threads for `sbalazs`, and 10 threads for `abouchez` because it was giving the best results for each program on our PC. |
| 83 | + |
| 84 | +Apart from the obvious global "wall" time reduction (`real` numbers), the raw parsing and data gathering in the threads match the number of threads and the running time (`user` numbers), and no syscall is involved by `abouchez` thanks to the memory mapping of the whole file (`sys` numbers, which contain only memory page faults). |
| 85 | + |
| 86 | +The `memmap()` feature makes the initial/cold `abouchez` call slower, because it needs to cache all measurements data from file into RAM (I have 32GB of RAM, so the whole data file will remain in memory, as on the benchmark hardware): |
| 87 | +``` |
| 88 | +ab@dev:~/dev/github/1brc-ObjectPascal/bin$ time ./abouchez measurements.txt -t=10 >resmrel4.txt |
| 89 | +
|
| 90 | +real 0m6,042s |
| 91 | +user 0m53,699s |
| 92 | +sys 0m2,941s |
| 93 | +``` |
| 94 | +This is the expected behavior, and will be fine with the benchmark challenge, which ignores the min and max values during its 10 times run. So the first run will just warm up the file into memory. |
| 95 | + |
| 96 | +On my Intel 13h gen processor with E-cores and P-cores, forcing thread to core affinity does not help: |
| 97 | +``` |
| 98 | +ab@dev:~/dev/github/1brc-ObjectPascal/bin$ ./abouchez measurements.txt -t=10 -v |
| 99 | +Processing measurements.txt with 10 threads and affinity=false |
| 100 | +result hash=8A6B746A,, result length=1139418, stations count=41343, valid utf8=1 |
| 101 | +done in 4.25s 3.6 GB/s |
| 102 | +ab@dev:~/dev/github/1brc-ObjectPascal/bin$ ./abouchez measurements.txt -t=10 -v -a |
| 103 | +Processing measurements.txt with 10 threads and affinity=true |
| 104 | +result hash=8A6B746A, result length=1139418, stations count=41343, valid utf8=1 |
| 105 | +done in 4.42s 3.5 GB/s |
| 106 | +``` |
| 107 | +Affinity may help on Ryzen 9, because its Zen 3 architecture is made of identical 16 cores with 32 threads, not this Intel E/P cores mess. But we will validate that on real hardware - no premature guess! |
| 108 | + |
| 109 | +The `-v` verbose mode makes such testing easy. The `hash` value can quickly check that the generated output is correct, and that it is valid `utf8` content (as expected). |
| 110 | + |
| 111 | +## Benchmark Integration |
| 112 | + |
| 113 | +Every system is quite unique, especially about its CPU multi-thread abilities. For instance, my Intel Core i5 has both P-cores and E-cores so its threading model is pretty unfair. The Zen architecture should be more balanced. |
| 114 | + |
| 115 | +So we first need to find out which options leverage at best the hardware it runs on. |
| 116 | + |
| 117 | +On the https://github.com/gcarreno/1brc-ObjectPascal challenge hardware, which is a Ryzen 9 5950x with 16 cores / 32 threads and 64MB of L3 cache, each thread using around 2.5MB of its own data, we should try several options with 16-24-32 threads, for instance: |
| 118 | + |
| 119 | +``` |
| 120 | +time ./abouchez measurements.txt -v -t=8 |
| 121 | +time ./abouchez measurements.txt -v -t=16 |
| 122 | +time ./abouchez measurements.txt -v -t=24 |
| 123 | +time ./abouchez measurements.txt -v -t=32 |
| 124 | +time ./abouchez measurements.txt -v -t=16 -a |
| 125 | +time ./abouchez measurements.txt -v -t=24 -a |
| 126 | +time ./abouchez measurements.txt -v -t=32 -a |
| 127 | +``` |
| 128 | +Please run those command lines, to guess which parameters are to be run for the benchmark, and would give the best results on the actual benchmark PC with its Ryzen 9 CPU. We will see if core affinity makes a difference here. |
| 129 | + |
| 130 | +Then we could run: |
| 131 | +``` |
| 132 | +time ./abouchez measurements.txt -v -t=1 |
| 133 | +``` |
| 134 | +This `-t=1` run is for fun: it will run the process in a single thread. It will help to guess how optimized (and lockfree) our parsing code is, and to validate the CPU multi-core abilities. In a perfect world, other `-t=##` runs should stand for a perfect division of `real` time per the number of working threads, and the `user` value reported by `time` should remain almost the same when we add threads up to the number of CPU cores. |
| 135 | + |
| 136 | +## Feedback Needed |
| 137 | + |
| 138 | +Here we will put some additional information, once our proposal has been run on the benchmark hardware. |
| 139 | + |
| 140 | +Stay tuned! |
| 141 | + |
| 142 | +## Ending Note |
| 143 | + |
| 144 | +There is a "*pure mORMot*" name lookup version available if you undefine the `CUSTOMHASH` conditional, which is around 40% slower, because it needs to copy the name into the stack before using `TDynArrayHashed`, and has a little more overhead. |
| 145 | + |
| 146 | +Arnaud :D |
0 commit comments