🐛Bugfix/644/listing makes pennsieve client fail

[sanderegg]

What do these changes do?

The connection with Pennsieve tends to fail after several calls to large datasets fail. This is an attempt to fix the problem.

• remove dependency to pennsieve client, now authenticating directly using pennsieve recipes and boto3/raw client
• remove dependency to pennsieve agent since we do not upload at the moment
• upgrades to later best practices in datcore-adapter fastapi application
• added 5 minutes caching to list files in a dataset
• increased default timeout in storage/webserver clients since datasets in pennsieve tends to be large (a better fix would be using long running tasks and/or not listing everything inside a dataset as this is rather costly in performance)
• now using cancel_on_disconnect decorator from servicelib instead of locally defined one
• fixed loglevel parsing, added classic middlewares
• added Makefile recipe to do real unit testing against pennsieve.io API

Bonus:

• aiohttp now reports 504 Gateway Timeout error when the internal client times-out on a forwarded call to storage or another service instead of 500.

Related issue/s

related to ITISFoundation/osparc-issues#644
related to ITISFoundation/dockerfiles#72

How to test

Checklist

[codecov]

Codecov Report

Merging #3464 (845cade) into master (8a9cb1e) will increase coverage by 2.0%.
The diff coverage is 87.4%.

@@           Coverage Diff            @@
##           master   #3464     +/-   ##
========================================
+ Coverage    83.3%   85.4%   +2.0%     
========================================
  Files         829     645    -184     
  Lines       35162   30044   -5118     
  Branches      739     552    -187     
========================================
- Hits        29321   25665   -3656     
+ Misses       5654    4231   -1423     
+ Partials      187     148     -39     

Flag	Coverage Δ
integrationtests	67.0% <75.0%> (-0.8%)	⬇️
unittests	82.0% <87.4%> (+1.7%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...library/src/servicelib/aiohttp/rest_middlewares.py	74.1% <25.0%> (-1.9%)	⬇️
...service_storage/datcore_adapter/datcore_adapter.py	35.0% <25.0%> (-0.1%)	⬇️
...e_service_datcore_adapter/api/errors/http_error.py	53.8% <33.3%> (-4.5%)	⬇️
...rage/datcore_adapter/datcore_adapter_exceptions.py	69.2% <66.6%> (-0.8%)	⬇️
...rage/src/simcore_service_storage/utils_handlers.py	89.2% <66.6%> (-2.8%)	⬇️
...mcore_service_datcore_adapter/modules/pennsieve.py	92.9% <83.0%> (+0.1%)	⬆️
...dels-library/src/models_library/app_diagnostics.py	100.0% <100.0%> (ø)
...ce_datcore_adapter/api/dependencies/application.py	100.0% <100.0%> (ø)
...vice_datcore_adapter/api/dependencies/pennsieve.py	100.0% <100.0%> (ø)
...imcore_service_datcore_adapter/api/module_setup.py	100.0% <100.0%> (ø)
... and 207 more

[pcrespov]

👍

You already did some useful upgrades on this service. Can I persuade you to do one extra "maintenance" PR follow this one and

• Add typecheck step (see unit-test-postgres-database for referece)
• Do full update of reqs. (some constraints were added due to pennsieve lib)

I suggest them in a separate PR for clarity

[pcrespov]

so these handlers are not anymore used? I do not see a replacement

[sanderegg]

no they are removed. and as a matter of fact they were never used.

[sanderegg]

+1

You already did some useful upgrades on this service. Can I persuade you to do one extra "maintenance" PR follow this one and

• Add typecheck step (see unit-test-postgres-database for referece)
• Do full update of reqs. (some constraints were added due to pennsieve lib)

I suggest them in a separate PR for clarity

@pcrespov 👍

1. requirements are already on the latest as far as I can tell
2. added the typechecking...

[GitHK]

Nice.
Please find some questions below.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication