⬆️ prune dependencies of simcore-sdk package and downstream services

[pcrespov]

What do these changes do?

Libraries evolve ans some of the dependencies are not anymore in use but remain listed in their requirements file. This is one of many maintenance PR to prune these deprecated dependencies.

We start with packages/simcore-sdk. With the help of pipreqs we discover which dependencies are actually in use simcore-sdk and comparing with packages/simcore-sdk/requirements/_base.in revealed deprecated ones (see changes in _base.in to detect which were actually pruned)
Then simcore-sdk's pruned reqs was propagated downstream (only the changes in simcore-sdk!) to services like

• api-server
• director-v2
• dynamic-sidecar
• swarm-deploy

This exercised also revealed:

• that some repo constraints (i.e. requirements/constraints.txt) were added and not propagated downstream. For that reason, some of the services above added some more changes. Among them, an incompatibility between the latest aiohttp and aiopg due to impossible constraints on the async-timeout library. Added to constraint.txt to all services using both libraries.
• ♻️ refactored
  • removed gc from tests/integration/02/test_computation.py::test_start_pipeline
  • added more log info to gc

Steps to prune a package and all services downstream

• pip install pipreqs
• find dependencies in use: pipreqs packages/simcore-sdk/src/simcore_sdk produces a requirements.txt file
• compare this file with packages/simcore-sdk/requirements/_base.in and remove unused libraries.
• dependencies with other libraries in the repo under packages has to be reviewed manually
• make full upgrade of reqs : make touch reqs
• Now it is time to check downstream dependencies : check for /packages/simcore-sdk/requirements/_base.in in all the repo including only *.in files.
• Go to those services requirements folder
• We want to selectively upgrade the pruning of the upstream library in this service. For that, we need to do a selective upgrade on any of the libraries that the upstream dependency includes, e.g. aiohttp for this example.

make touch 
make reqs upgrade=aiohttp

• That should do a selective upgrade of the service

Related issue/s

ITISFoundation/osparc-issues#428

How to test

all tests affected

[codecov]

Codecov Report

Merging #2664 (9504636) into master (a357669) will decrease coverage by 0.0%.
The diff coverage is 94.4%.

@@           Coverage Diff            @@
##           master   #2664     +/-   ##
========================================
- Coverage    78.1%   78.0%   -0.1%     
========================================
  Files         636     636             
  Lines       26060   26064      +4     
  Branches     2524    2524             
========================================
- Hits        20363   20353     -10     
- Misses       5032    5044     +12     
- Partials      665     667      +2     

Flag	Coverage Δ
integrationtests	66.7% <91.6%> (-0.2%)	⬇️
unittests	73.6% <83.3%> (-0.1%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ce_webserver/resource_manager/garbage_collector.py	75.2% <91.6%> (-0.9%)	⬇️
...ibrary/src/servicelib/aiohttp/application_setup.py	82.4% <100.0%> (+0.8%)	⬆️
..._director_v2/modules/dynamic_sidecar/client_api.py	68.7% <0.0%> (-1.9%)	⬇️
...tor_v2/modules/dynamic_sidecar/scheduler/events.py	93.6% <0.0%> (-1.5%)	⬇️
.../src/simcore_service_webserver/director_v2_core.py	65.3% <0.0%> (-1.2%)	⬇️
.../director/src/simcore_service_director/producer.py	61.0% <0.0%> (-0.9%)	⬇️
...ector_v2/modules/dynamic_sidecar/scheduler/task.py	85.9% <0.0%> (-0.6%)	⬇️

[GitHK]

Very nice!
Could you please add the description of this PR as notes in the repository? It will be useful in the future to prune dependencies.

[sanderegg]

cool, that is a nice tool.

Now concerning the instructions to do this, don't you think this should go in some markdown file in the repo? like that it will go into the limbos of github.