Packaging for NixOS

[CMCDragonkai]

I'm currently working on packaging for NixOS: NixOS/nixpkgs#49438

I may ask for help regarding the integration.

[efiop]

Hi @CMCDragonkai !

Great stuff! Thank you for your work! Please don't hesitate to ask any questions, we'll be happy to help 🙂

[efiop]

@CMCDragonkai Also note that we have binary packages as well. For example, our deb and rpm packages contain dvc built with pyinstaller with all of the possible dependencies, so it doesn't rely on anything on the system and carries all the libs(including python itself) with it. You should be able to utilize that as well, if you are not keen on packaging every pip dependency separately. See scripts/build_posix.sh for more info.

[CMCDragonkai]

Nix prefers to get everything built from source barring proprietary applications.

[CMCDragonkai]

I've packaged all the dependencies for dvc into NixOS. Now I'm onto packaging dvc itself. However your pip installation doesn't have a source distribution. NixOS prefers to build things from the source. Can you push up your source distributions to PyPi as well? https://pypi.org/project/dvc/#files

Note that source distribution should contain any tests for dvc as well.

@CMCDragonkai , what if instead of fetching from pypi you set something like:

src = fetchurl {
  url = "https://github.com/iterative/dvc/archive/${pkgver}.tar.gz";
  sha256 = "sha of the tar.gz";
};

I don't know if it's possible, but that's what I'm currently doing for the Arch build: PKGBUILD.

However, I think it is a good idea to release the source code to pypi

By the way, great work so far, @CMCDragonkai ! I noticed that you also packaged th missing dependencies on nixpkgs 👍 !
Regarding your question about the optional dependencies, I've seen both enableX and supportX (e.g. supportSSH)

NOTE: remember to include HDFS as a remote

[CMCDragonkai]

I can use fetchFromGithub instead that's usually the standard as long as Github releases are considered as official as PyPI releases then it's fine.

Yes, you can rely on GitHub releases, as far as I know, we are supporting them 😃

[CMCDragonkai]

Ok building from source results in:

»» ~/Projects/nixpkgs
 ♜ nix-build ./default.nix --attr 'dvc'      (dvc) pts/8 17:30:56
these derivations will be built:
  /nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv
building '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv'...
unpacking sources
unpacking source archive /nix/store/wj4qqj4qvinvrgm1njq65klfdh6ngzx6-source
source root is source
setting SOURCE_DATE_EPOCH to timestamp 315619200 of file source/tests/utils/httpd.py
patching sources
configuring
building
Traceback (most recent call last):
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 83, in <module>
    refresh()
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 73, in refresh
    if not Git.refresh(path=path):
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/cmd.py", line 290, in refresh
    raise ImportError(err)
ImportError: Bad git executable.
The git executable must be specified in one of the following ways:
    - be included in your $PATH
    - be set via $GIT_PYTHON_GIT_EXECUTABLE
    - explicitly set via git.refresh()

All git commands will error until this is rectified.

This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
    - quiet|q|silence|s|none|n|0: for no warning or exception
    - warn|w|warning|1: for a printed warning
    - error|e|raise|r|2: for a raised exception

Example:
    export GIT_PYTHON_REFRESH=quiet


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 5, in <module>
    from dvc import VERSION
  File "/build/source/dvc/__init__.py", line 53, in <module>
    __version__ = _update_version_file()
  File "/build/source/dvc/__init__.py", line 20, in _update_version_file
    from git import Repo
  File "/nix/store/y2ck276xvrpz7l6yghzyx1qy2kyw19aq-python3.7-GitPython-2.1.11/lib/python3.7/site-packages/git/__init__.py", line 85, in <module>
    raise ImportError('Failed to initialize: {0}'.format(exc))
ImportError: Failed to initialize: Bad git executable.
The git executable must be specified in one of the following ways:
    - be included in your $PATH
    - be set via $GIT_PYTHON_GIT_EXECUTABLE
    - explicitly set via git.refresh()

All git commands will error until this is rectified.

This initial warning can be silenced or aggravated in the future by setting the
$GIT_PYTHON_REFRESH environment variable. Use one of the following values:
    - quiet|q|silence|s|none|n|0: for no warning or exception
    - warn|w|warning|1: for a printed warning
    - error|e|raise|r|2: for a raised exception

Example:
    export GIT_PYTHON_REFRESH=quiet

builder for '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv' failed with exit code 1
error: build of '/nix/store/rs84jf5z5bmra71ps6ikaw1jx9jp2wvh-dvc-0.24.0.drv' failed

The tests require the existence of git. So I add that into buildInputs.

Furthermore the tests also require nosetests. But not available.

[CMCDragonkai]

It appears that test dependencies need quite a lot more stuff: https://github.com/iterative/dvc/blob/master/test-requirements.txt

I'd like to know more about your test enviornment. Even after adding all the dependencies, the first error I get is:

Error: Failed to load dir cache '../../b/uild/dvc-test.4363.yk9hglkp.0d76514b-9d20-44bd-a35e-0abe6b4b2720/.dvc/cache/12/3.dir' - [Errno 2] No such file or directory: '/b/uild/dvc-test.4363.yk9hglkp.0d76514b-9d20-44bd-a35e-0abe6b4b2720/.dvc/cache/12/3.dir'

[CMCDragonkai]

I just realised that some of the tests call awscli which means it relies on test AWS services. Does this mean the tests are meant to be executed with DVC's private testing servers? In that case is there a way to only test dvc with unattended tests? If not, I'll have to disable tests.

[CMCDragonkai]

Another question, is that when I have finally built the dvc script. Running the script results in this message saying:

/nix/store/ydk0mfpvn9smcmn72wc9i20slv1d2b79-python3-3.7.2/bin/python3.7: No module named dvc

For simplicity, @CMCDragonkai, I would recommend you to disable the tests (we are already testing the code on our CI, so we are covered), we can enable them after fixing those dependency issues.

About the No module named dvc error, is hard to tell 🤔
Do you have more details about it? What about running a shell and import dvc

[CMCDragonkai]

The PR for Nixpkgs is here: NixOS/nixpkgs#54530

I've disabled all tests. So the last thing to figure out is that No module named dvc.