Conversation
gcc makes template static data members and static local variables in inline functions globally unique. This is, however, a problem for kernels, since two kernels or kernel versions could use a static variable with the same name, resulting in inconsistent state. Note that this is not resolved by `RTLD_LOCAL` being used when `dlopen`'ing Python extensions. These globally unique symbols are excepted in hiding by `RTLD_LOCAL`. We resolve this by passing `-fno-gnu-unique` to compilers that support it (compilers that don't support it, most likely do not make these symbols globally unique).
Seems like we got it implicitly on x86_64-linux.
Coverage report —
|
| Name | Stmts | Miss | Cover | Missing |
|---|---|---|---|---|
| src/kernels/__init__.py | 10 | 0 | 100% | |
| src/kernels/_system.py | 6 | 1 | 83% | 10 |
| src/kernels/_versions.py | 63 | 7 | 89% | 46, 49, 52-53, 56-57, 100 |
| src/kernels/backends.py | 194 | 55 | 72% | 40, 44, 48-51, 68, 90, 108, 117, 121, 125-127, 148, 170, 181, 188-191, 201, 205-225, 233, 256-276 |
| src/kernels/compat.py | 8 | 1 | 88% | 5 |
| src/kernels/deps.py | 54 | 4 | 93% | 58-59, 95, 98 |
| src/kernels/layer/__init__.py | 6 | 0 | 100% | |
| src/kernels/layer/_interval_tree.py | 103 | 4 | 96% | 23, 52, 147, 150 |
| src/kernels/layer/device.py | 48 | 14 | 71% | 42, 47-49, 91, 96-98, 101, 149, 152, 155-157 |
| src/kernels/layer/func.py | 81 | 7 | 91% | 81, 111, 183, 301, 307, 320, 338 |
| src/kernels/layer/globals.py | 5 | 0 | 100% | |
| src/kernels/layer/kernelize.py | 74 | 8 | 89% | 255, 281, 289-290, 296, 300, 316-318 |
| src/kernels/layer/layer.py | 210 | 16 | 92% | 167, 210, 216, 229, 337, 417-418, 430, 439, 447, 458, 487, 491, 504, 557, 587 |
| src/kernels/layer/mode.py | 14 | 0 | 100% | |
| src/kernels/layer/repos.py | 130 | 34 | 74% | 27, 33, 36-41, 61-62, 68, 71-74, 88, 92, 101-102, 108, 111-114, 121-122, 128, 131-134, 141-142, 148, 151-154, 235 |
| src/kernels/lockfile.py | 71 | 46 | 35% | 37-104, 108-131 |
| src/kernels/status.py | 49 | 2 | 96% | 23, 81 |
| src/kernels/utils.py | 313 | 55 | 82% | 65, 77-81, 87-88, 246, 250, 253, 315, 323, 362-363, 401, 432, 437, 472, 701, 704, 706, 712, 725-726, 747-759, 763-770, 778, 782-792, 796-803, 841, 845, 864, 866 |
| src/kernels/variants.py | 270 | 22 | 92% | 63, 94, 115, 145, 254-255, 297-300, 302, 386-393, 399-405, 436-442, 454-460, 559-561 |
| src/kernels/verify.py | 88 | 1 | 99% | 32 |
| TOTAL | 1797 | 277 | 85% |
Updated by the Test kernels workflow on commit 0abe75297f898f5ee21e687c2c48ce08e95ae6ba.
| @@ -0,0 +1,41 @@ | |||
| name: "Build and test kernel (CPU)" | |||
There was a problem hiding this comment.
Could I get an explanation on why it could not be merged into our existing workflows?
There was a problem hiding this comment.
We have separate workflows for CUDA, ROCm, and XPU. CPU was just missing. There are some CPU-only things in the CUDA tests, but I also want to move them into this workflow in the future. The CPU workflow and CI target is much nicer, because we can run tests during the build (which we can't for CUDA, due to the sandbox).
| # CPU kernels don't require special hardware, so we build and run | ||
| # their CI tests (e.g. the symbol-conflicts pytest suite) in one step. | ||
| - name: Build and test all CPU example kernels | ||
| run: nix build -L ./examples/kernels#ci-build-cpu |
There was a problem hiding this comment.
Should use kernel-builder ... here?
There was a problem hiding this comment.
These are Nix outputs that build multiple kernels. Not something kernel-builder can build.
| USE_SABI 3 | ||
| WITH_SOABI) | ||
|
|
||
| # Avoid that definitions of template static data members and static local |
There was a problem hiding this comment.
We should probably take an example kernel where this was surfaced?
There was a problem hiding this comment.
Not sure. I think the example kernels that are included in this PR describe the issue in a nice and minimalist way?
2 participants
gcc makes template static data members and static local variables in inline functions globally unique. This is, however, a problem for kernels, since two kernels or kernel versions could use a static variable with the same name, resulting in inconsistent state.
Note that this is not resolved by
RTLD_LOCALbeing used whendlopen'ing Python extensions. These globally unique symbols are excepted in hiding byRTLD_LOCAL.We resolve this by passing
-fno-gnu-uniqueto compilers that support it (compilers that don't support it, most likely do not make these symbols globally unique).Also add tests that would catch such unwanted sharing in the future.