Test packages

[romainx]

This test module tests if R and Python packages installed can be imported.
It's a basic test aiming to prove that the package is working properly.

The goal is to detect import errors that can be caused by incompatibilities between packages for example:

• datascience-notebook: newer images generate " Illegal instruction (core dumped)" when importing sympy #1012: issue importing sympy
• pyspark: undefined symbol in pyarrow #966: isssue importing pyarrow

This module checks dynmamically, through the CondaPackageHelper, only the specified packages i.e. packages requested by conda install in the Dockerfiles.
This means that it does not check dependencies. This choice is a tradeoff to cover the main requirements while achieving reasonabe test duration.
However it could be easily changed (or completed) to cover also dependencies package_helper.installed_packages() instead of package_helper.specified_packages().

Example:

$ make test/datascience-notebook

# [...]
# test/test_packages.py::test_python_packages 
# --------------------------------------------------------------------------------------------- live log setup ----------------------------------------------------------------------------------------------
# 2020-03-08 09:56:04 [    INFO] Starting container jupyter/datascience-notebook ... (helpers.py:51)
# 2020-03-08 09:56:04 [    INFO] Running jupyter/datascience-notebook with args {'detach': True, 'ports': {'8888/tcp': 8888}, 'tty': True, 'command': ['start.sh', 'bash', '-c', 'sleep infinity']} ... (conftest.py:78)
# 2020-03-08 09:56:04 [    INFO] Grabing the list of specifications ... (helpers.py:76)
# ---------------------------------------------------------------------------------------------- live log call ----------------------------------------------------------------------------------------------
# 2020-03-08 09:56:07 [    INFO] Testing the import of packages ... (test_packages.py:125)
# 2020-03-08 09:56:07 [    INFO] Trying to import conda (test_packages.py:127)
# 2020-03-08 09:56:07 [    INFO] Trying to import notebook (test_packages.py:127)
# 2020-03-08 09:56:08 [    INFO] Trying to import jupyterhub (test_packages.py:127)
# [...]

[parente]

Love it! ❤️

I left a couple doc string change suggestions and one comment about what to test inline.

I wonder what your thoughts are on what we do in the case where a test fails because of a problem with a single upstream package, but that the rest of packages across image(s) all work correctly. Do we hold up releasing any further changes in the images here until we resolve the problem upstream? Do we choose to ignore the failing test and continue releasing unrelated changes here while building images with clearly broken dependencies?

[romainx]

@parente Thanks for the positive feedback and for the relevant review. 😄

To answer your question, my opinion is to fail if an import error is raised -- if it's not the case users may have a problem and could open an issue. However in some case it could be interesting to ignore some errors. To be ready to do it, I have modified the code to introduce a max_failures parameter.
The parameter is equal to zero by default and can be tuned. If some errors are raised but ignored a warning is printed.

if len(failures) > max_failures:
    raise AssertionError(failures)
elif len(failures) > 0:
    LOGGER.warning(f"Some import(s) has(have) failed: {failures}")

I have committed all the change except the parsing of the package bug ... So I'm flagging it as WIP.

See you.

[romainx]

@parente I'm waiting for the CI to finish, but I think it's OK now.
If you're agree, we are ready to merge 🎉

[parente]

Let's give it a shot! Thanks for putting this testing in place.