python3Packages.ffcv: init at 0.0.3

[samuela]

Motivation for this change

Introduce python3Packages.ffcv. FFCV is an exciting new data loading library for computer vision applications.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandbox = true set in nix.conf? (See Nix manual)
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 22.05 Release Notes (or backporting 21.11 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
  • (Release notes changes) Ran nixos/doc/manual/md-to-db.sh to update generated release notes
• Fits CONTRIBUTING.md.

[samuela]

It's almost working. The package includes a C++ bit that is compiled and linked via distutils called ffcv._libffcv. That builds fine, but the tests don't seem to find it in PYTHONPATH:

____________________ ERROR collecting tests/test_writer.py _____________________
ImportError while importing test module '/build/source/tests/test_writer.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/nix/store/afi0ysqw20yiiw2gr2d28dx40bc4ddf8-python3-3.9.10/lib/python3.9/importlib/__init__.py:127: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
...
ffcv/libffcv.py:5: in <module>
    import ffcv._libffcv
E   ModuleNotFoundError: No module named 'ffcv._libffcv'

Anyone have any idea what could be going on here? IIRC @jonringer you mentioned something about this (or maybe it was cython?) a while back?

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/packaging-a-python-library-with-c-distutils-extension/17732/1

[SuperSandro2000]

I am not sure if pkgconfig the python package should go to nativeBuildInputs.

[samuela]

Do you mean that I should move it to nativeBuildInputs? Or something else?

[SuperSandro2000]

Yeah, I think we want to move pkgconfig into nativeBuildInputs.

[samuela]

Moving pkgconfig into nativeBuildInputs doesn't work unfortunately:

Executing setuptoolsBuildPhase
Traceback (most recent call last):
  File "/build/source/nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 30, in <module>
    extension_kwargs = pkgconfig('opencv4', extension_kwargs)
  File "setup.py", line 18, in pkgconfig
    raise Exception()
Exception
builder for '/nix/store/zsvvs4ispid5s6viwnpvcfl12p3hmzvl-python3.9-ffcv-0.0.3.drv' failed with exit code 1

[samuela]

@SuperSandro2000 any idea how to deal with the ffcv._libffcv situation?

[SuperSandro2000]

@SuperSandro2000 any idea how to deal with the ffcv._libffcv situation?

I think you want something similar to export PYTHONPATH="$out/${python.sitePackages}:$PYTHONPATH" in the prePhase where the error happens.

[samuela]

Adding

  preCheck = ''
    export PYTHONPATH="$out/${python.sitePackages}:$PYTHONPATH"
  '';

still gives me the same errors unfortunately.

[samuela]

Weirdly enough, $out/${python.sitePackages} is /nix/store/n6ck59zm07s8qk4j1szwlq9jys35n7ks-python3.9-ffcv-0.0.3/lib/python3.9/site-packages and the file does exist in there: $out/${python.sitePackages}/ffcv/_libffcv.cpython-39-x86_64-linux-gnu.so. I guess for some reason it can't be loaded in the nix build environment... possibly because it tries to access CUDA stuff?

[samuela]

Ok well I updated everything based on the feedback thus far in 00dc6dd. I'll mark as "ready for review" and merge tomorrow unless anyone objects.

[nixos-discourse]

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/packaging-a-python-library-with-c-distutils-extension/17732/5