Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add various Music Information Retrieval python libraries #82892

Closed
wants to merge 7 commits into from

Conversation

@jmorag
Copy link

jmorag commented Mar 18, 2020

Motivation for this change

Expand nixpkgs to include pypi libraries relevant to music information retrieval.

Things done

Added mirdata, mir_eval, music21, and jams packages.

  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.
@jmorag jmorag requested review from FRidh and jonringer as code owners Mar 18, 2020
Copy link
Contributor

jonringer left a comment

thank you for opening your first PR

please try to apply the suggestions to the other packages

maintainers/maintainer-list.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/jams/default.nix Outdated Show resolved Hide resolved
pkgs/development/python-modules/music21/default.nix Outdated Show resolved Hide resolved
mirdata = callPackage ../development/python-modules/mirdata { };

mir_eval = callPackage ../development/python-modules/mir_eval { };

jams = callPackage ../development/python-modules/jams { };

music21 = callPackage ../development/python-modules/music21 { };

Comment on lines 3278 to 3285

This comment has been minimized.

Copy link
@jonringer

jonringer Mar 18, 2020

Contributor

please attempt to sort these amongst the other packages

@jmorag jmorag force-pushed the jmorag:master branch from 2955767 to 83379ec Mar 19, 2020
@jmorag
Copy link
Author

jmorag commented Mar 19, 2020

Mirdata tests fail with

Traceback (most recent call last):
  File "nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 42, in <module>
    'dali': ['DALI-dataset>=1.0.0'],
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 238, in run
    self.run_tests()
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 260, in run_tests
    exit=False,
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 124, in parseArgs
    self._do_discovery(argv[2:])
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 244, in _do_discovery
    self.createTests(from_discovery=True, Loader=Loader)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 154, in createTests
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 349, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 406, in _find_tests
    full_path, pattern, namespace)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 483, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/build/mirdata-0.1.2/tests/test_utils.py", line 7, in <module>
    from mirdata import utils
  File "/build/mirdata-0.1.2/mirdata/__init__.py", line 7, in <module>
    from . import beatles
  File "/build/mirdata-0.1.2/mirdata/beatles.py", line 20, in <module>
    import librosa
  File "/nix/store/242byyk84j3r3c878v53fjxkvxh5qdpa-python3.7-librosa-0.7.1/lib/python3.7/site-packages/librosa/__init__.py", line 12, in <module>
    from . import core
  File "/nix/store/242byyk84j3r3c878v53fjxkvxh5qdpa-python3.7-librosa-0.7.1/lib/python3.7/site-packages/librosa/core/__init__.py", line 123, in <module>
    from .time_frequency import *  # pylint: disable=wildcard-import
  File "/nix/store/242byyk84j3r3c878v53fjxkvxh5qdpa-python3.7-librosa-0.7.1/lib/python3.7/site-packages/librosa/core/time_frequency.py", line 12, in <module>
    from ..util.deprecation import Deprecated
  File "/nix/store/242byyk84j3r3c878v53fjxkvxh5qdpa-python3.7-librosa-0.7.1/lib/python3.7/site-packages/librosa/util/__init__.py", line 77, in <module>
    from .utils import *  # pylint: disable=wildcard-import
  File "/nix/store/242byyk84j3r3c878v53fjxkvxh5qdpa-python3.7-librosa-0.7.1/lib/python3.7/site-packages/librosa/util/utils.py", line 1825, in <module>
    def __shear_dense(X, factor=+1, axis=-1):
  File "/nix/store/05jp6hjww0a5s7k01b6hph1iqmzmjjqg-python3.7-numba-0.48.0/lib/python3.7/site-packages/numba/decorators.py", line 193, in wrapper
    disp.enable_caching()
  File "/nix/store/05jp6hjww0a5s7k01b6hph1iqmzmjjqg-python3.7-numba-0.48.0/lib/python3.7/site-packages/numba/dispatcher.py", line 679, in enable_caching
    self._cache = FunctionCache(self.py_func)
  File "/nix/store/05jp6hjww0a5s7k01b6hph1iqmzmjjqg-python3.7-numba-0.48.0/lib/python3.7/site-packages/numba/caching.py", line 614, in __init__
    self._impl = self._impl_class(py_func)
  File "/nix/store/05jp6hjww0a5s7k01b6hph1iqmzmjjqg-python3.7-numba-0.48.0/lib/python3.7/site-packages/numba/caching.py", line 349, in __init__
    "for file %r" % (qualname, source_path))
RuntimeError: cannot cache function '__shear_dense': no locator available for file '/nix/store/242byyk84j3r3c878v53fjxkvxh5qdpa-python3.7-librosa-0.7.1/lib/python3.7/site-packages/librosa/util/utils.py'
builder for '/nix/store/1yzkzqkjibjgidp31zaw7hy8n642iz01-python3.7-mirdata-0.1.2.drv' failed with exit code 1
cannot build derivation '/nix/store/r6gcp3kfqnpzkkhlxk1nmxw81x1gaxqf-python3-3.7.6-env.drv': 1 dependencies couldn't be built
error: build of '/nix/store/r6gcp3kfqnpzkkhlxk1nmxw81x1gaxqf-python3-3.7.6-env.drv' failed

It looks like it has something to do with librosa/librosa#1013 but I don't know how to emulate the solution in nix.

@jmorag
Copy link
Author

jmorag commented Mar 19, 2020

Music21 tests fail with

Traceback (most recent call last):
  File "nix_run_setup", line 8, in <module>
    exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\\r\\n', '\\n'), __file__, 'exec'))
  File "setup.py", line 71, in <module>
    include_package_data=True,
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/__init__.py", line 144, in setup
    return distutils.core.setup(**attrs)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 238, in run
    self.run_tests()
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 260, in run_tests
    exit=False,
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 124, in parseArgs
    self._do_discovery(argv[2:])
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 244, in _do_discovery
    self.createTests(from_discovery=True, Loader=Loader)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/main.py", line 154, in createTests
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 349, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 406, in _find_tests
    full_path, pattern, namespace)
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 483, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/nix/store/110cqfw7ncbn4haynn8x8ziwzmr55dlg-python3.7-setuptools-45.2.0/lib/python3.7/site-packages/setuptools/command/test.py", line 55, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/nix/store/xb1mp0f16lv23rwhfdc1665v5r20jfmm-python3-3.7.6/lib/python3.7/unittest/loader.py", line 154, in loadTestsFromName
    module = __import__(module_name)
  File "/build/music21-5.7.2/music21/ext/joblib/externals/cloudpickle/cloudpickle_fast.py", line 388, in <module>
    class CloudPickler(Pickler):
  File "/build/music21-5.7.2/music21/ext/joblib/externals/cloudpickle/cloudpickle_fast.py", line 410, in CloudPickler
    dispatch[types.CellType] = _cell_reduce
AttributeError: module 'types' has no attribute 'CellType'

which I think has to do with a python2/3 incompatibility.

@jmorag
Copy link
Author

jmorag commented Mar 19, 2020

Jams tests fail when trying to import jams.version but there aren't any tests anyway.

@jmorag
Copy link
Author

jmorag commented Mar 19, 2020

I've force-pushed the other suggested changes. Thank you for your feedback.

Copy link
Member

FRidh left a comment

Are you actually going to use all these libraries? We should not just add packages to Nixpkgs for the sake of it, as it does add to the maintenance burden.

pkgs/development/python-modules/mir_eval/default.nix Outdated Show resolved Hide resolved
@jmorag jmorag force-pushed the jmorag:master branch from 83379ec to 94a80e2 Mar 19, 2020
@jmorag jmorag force-pushed the jmorag:master branch from 94a80e2 to 7609a91 Mar 19, 2020
@jmorag
Copy link
Author

jmorag commented Mar 19, 2020

Are you actually going to use all these libraries? We should not just add packages to Nixpkgs for the sake of it, as it does add to the maintenance burden.

As a music technology student, I do use these packages for coursework and research. If you don't want the added burden of maintaining hand-written pypi expressions I can just maintain them in my fork.

@drewrisinger
Copy link
Contributor

drewrisinger commented Mar 24, 2020

Another option is https://github.com/nix-community/NUR. They're not in the main nixpkgs repository, but almost as easy to use, and faster to get them out there than the Nix PR process.

@jonringer
Copy link
Contributor

jonringer commented Mar 24, 2020

true, also, if these are just plain python packages, it's enough to create a virtualenv/pipenv/conda env/poetry env with nix, and just jump into using normal python tooling to solve your needs.

Main reason to add python packages is so that python applications can be exported, or a python package that has some native dependencies which may not play nicely in a nix-shell

@ofborg ofborg bot removed the 6.topic: python label May 4, 2020
@drewrisinger
Copy link
Contributor

drewrisinger commented May 4, 2020

@jmorag did you decide what path you want to go with these packages? NUR, in nixpkgs, or self-maintained in fork? Can you close this PR if you're not using nixpkgs?

@jmorag
Copy link
Author

jmorag commented May 4, 2020

Self maintained in my fork.

@jmorag jmorag closed this May 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

4 participants
You can’t perform that action at this time.