Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add DistributedEnvironmentPlugin #258

Merged
merged 2 commits into from
Apr 15, 2022

Conversation

Andrew42
Copy link
Contributor

Addresses #219 by including a new plugins.py file that provides a NannyPlugin that can install a locally specified pip installable package. The plugin can also be provided a list of individual python files (e.g. analysis specific processor definitions) that should be included alongside the main package.

This PR does not currently include a corresponding pytest test as it is unclear how to best setup the inputs for a useful unit test. I plan to make a github issue, which will link back to this PR, regarding the need for such a test, so that we don't forget.

There are a few other parts of coffea-casa that I would like to get feedback on if they should also be modified in this PR. These are related to the changes needed to actually enable the use of NannyPlugins in general.

  1. The coffea-casa Dockerfile should presumably enable John's patch for dask.distributed.
  2. The patch file itself needs to be regenerated to work with newer versions of dask.distributed. We've already remade the patch during testing (e.g. here), but I'd like to get confirmation on which version of distributed is currently being used in coffea-casa to know if the updated patch should be used or not.
  3. coffea_casa.py has a number of nanny related lines commented out. I assume that we would want to uncomment these lines?

@oshadura
Copy link
Member

oshadura commented Apr 1, 2022

I see that GH Actions were not "happy":

============================= test session starts ==============================
platform linux -- Python 3.6.15, pytest-7.0.1, pluggy-1.0.0
rootdir: /home/runner/work/coffea-casa/coffea-casa, configfile: tox.ini, testpaths: tests
collected 0 items / 3 errors

==================================== ERRORS ====================================
__________________ ERROR collecting tests/test_coffeacasa.py ___________________
ImportError while importing test module '/home/runner/work/coffea-casa/coffea-casa/tests/test_coffeacasa.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_coffeacasa.py:7: in <module>
    from coffea_casa import CoffeaCasaCluster
coffea_casa/__init__.py:8: in <module>
    from .plugin import DistributedEnvironmentPlugin
coffea_casa/plugin.py:8: in <module>
    from distributed.diagnostics.plugin import NannyPlugin
E   ImportError: cannot import name 'NannyPlugin'
________________ ERROR collecting tests/test_coffeacasa_args.py ________________
ImportError while importing test module '/home/runner/work/coffea-casa/coffea-casa/tests/test_coffeacasa_args.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_coffeacasa_args.py:4: in <module>
    from coffea_casa import CoffeaCasaCluster
coffea_casa/__init__.py:8: in <module>
    from .plugin import DistributedEnvironmentPlugin
coffea_casa/plugin.py:8: in <module>
    from distributed.diagnostics.plugin import NannyPlugin
E   ImportError: cannot import name 'NannyPlugin'
____________________ ERROR collecting tests/test_config.py _____________________
ImportError while importing test module '/home/runner/work/coffea-casa/coffea-casa/tests/test_config.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
/opt/hostedtoolcache/Python/3.6.15/x64/lib/python3.6/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_config.py:3: in <module>
    import coffea_casa.config
coffea_casa/__init__.py:8: in <module>
    from .plugin import DistributedEnvironmentPlugin
coffea_casa/plugin.py:8: in <module>
    from distributed.diagnostics.plugin import NannyPlugin
E   ImportError: cannot import name 'NannyPlugin'
=========================== short test summary info ============================
ERROR tests/test_coffeacasa.py
ERROR tests/test_coffeacasa_args.py
ERROR tests/test_config.py
!!!!!!!!!!!!!!!!!!! Interrupted: 3 errors during collection !!!!!!!!!!!!!!!!!!!!

Can you take a look?

@oshadura oshadura merged commit 9db238e into CoffeaTeam:master Apr 15, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants