python3Packages.dask: fix sandboxed builds #120310
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
nbren12
force-pushed
the
fix-dask-in-sandbox
branch
from
1e393f1
to
031a8b7
Compare
10 tasks
|
Result of 7 packages marked as broken and skipped:
16 packages skipped due to time constraints:
46 packages built successfully:
1 suggestion:
|
|
Did you report the issue upstream? If not, please do so, submit the patch, and fetch it instead of including directly in Nixpkgs. |
|
I opened a PR upstream: dask/dask#7601. Will I need to wait for this to be merged before using it here? |
No need for that. Please use |
nbren12
force-pushed
the
fix-dask-in-sandbox
branch
from
031a8b7
to
379f74d
Compare
I just did this. One question: can github possibly garbage collect a patch if the upstream PR uses a merge strategy that modifies the history (e.g. rebase or squash)? |
Importing dask.dataframe in a sandboxed build results in a TypeError like
this:
File "/nix/store/nv60iri29bia4szhhcvsdxgsci4wxvp6-python3.8-dask-2021.03.0/lib/python3.8/site-packages/dask/dataframe/io/csv.py", line 392, in <module>
AUTO_BLOCKSIZE = auto_blocksize(TOTAL_MEM, CPU_COUNT)
File "/nix/store/nv60iri29bia4szhhcvsdxgsci4wxvp6-python3.8-dask-2021.03.0/lib/python3.8/site-packages/dask/dataframe/io/csv.py", line 382, in auto_blocksize
blocksize = int(total_memory // cpu_count / memory_factor)
TypeError: unsupported operand type(s) for //: 'int' and 'NoneType'
This occurs because dask.dataframe has a non-deterministic component which
generates an automatic chunk-size based on system information.
This went unnoticed because the dask tests were disabled.
Changes:
- add a patch making the chunk-size inference more robust
- re-enable the tests
Resolves NixOS#120307
nbren12
force-pushed
the
fix-dask-in-sandbox
branch
from
379f74d
to
ceeaf2d
Compare
It shouldn't https://github.community/t/does-github-ever-purge-commits-or-files-that-were-visible-at-some-time/1944/2. |
|
This is a semi-automatic executed nixpkgs-review with nixpkgs-review-checks extension. It is checked by a human on a best effort basis and does not build all packages (e.g. lumo, tensorflow or pytorch). Result of 7 packages marked as broken and skipped:
4 packages failed to build and already failed to build on hydra master:
1 package failed to build and are new build failure:
57 packages built:
|
10 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation for this change
Allow
dask.dataframeto be used in sandboxed builds. See #120307.Things done
sandboxinnix.confon non-NixOS linux)nix-shell -p nixpkgs-review --run "nixpkgs-review wip"./result/bin/)nix path-info -Sbefore and after)