Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

✨♻️🐛 serviceslib[fastapi] and projects tables validation scripts #2550

Merged
merged 35 commits into from
Sep 24, 2021

Conversation

pcrespov
Copy link
Member

@pcrespov pcrespov commented Sep 22, 2021

What do these changes do?

  • ✨ New diagnostics script: in models-library/scripts a script to validate db projects table exported to a CSV
  • ✨ Nex extra for servicelib[fastapi] package with fastapi-only features (released as servicelib 1.1.0)
    • adapts Makefile and CI
    • utils to override FastApi.openapi
    • extends functools
  • 🐛 FIXES openapi.json in catalog service
    • adds dependency to servicelib[fastapi]
    • splits requirements _base.*, _packages.*
  • 🐛 FiXES pylint tests
  • ♻️ cleanup pytest-simcore

NOTES

  • It is becoming very convoluted to deal with intra-dependencies using our current requirements approach. We should look for an simplified approach
  • running models-library/scripts/validate-pg-projects.py revealed:
    • Found 7/1532 invalid projects in aws-production
    • Found 7/1535 invalid projects in aws-staging
    • Found 40/776 invalid projects in dalco-production
    • Found 12/1163 invalid projects in dalgo-staging

Followup in #2552

Related issue/s

How to test

  • download CSV file for a projects table using Adminer export
  • Let's place them in packages/models-library/scripts/ignore.keep and run a validation script
$ make devenv
$ source .venv/bin/activate
$ cd packages/models-library
$ make install-dev
$ cd scripts
$ mkdir ignore.keep
# place in scripts/ignore.keep all csv files
$ for f in ignore.keep/*.csv; do python validate-pg-projects.py -v $f >$f.log 2>&1 ; done
  • open log files in ignore.keep/

@pcrespov pcrespov changed the title Is544/validate projects WIP: data validation and fastapi tooling Sep 22, 2021
@pcrespov pcrespov self-assigned this Sep 22, 2021
@pcrespov pcrespov added this to the Capra delle nevi milestone Sep 22, 2021
@codecov
Copy link

codecov bot commented Sep 22, 2021

Codecov Report

Merging #2550 (4b6e38f) into master (7162ba4) will increase coverage by 0.0%.
The diff coverage is 82.4%.

Impacted file tree graph

@@          Coverage Diff           @@
##           master   #2550   +/-   ##
======================================
  Coverage    76.9%   76.9%           
======================================
  Files         619     622    +3     
  Lines       23972   24028   +56     
  Branches     2345    2355   +10     
======================================
+ Hits        18438   18482   +44     
- Misses       4908    4918   +10     
- Partials      626     628    +2     
Flag Coverage Δ
integrationtests 65.4% <ø> (+<0.1%) ⬆️
unittests 71.4% <82.4%> (+<0.1%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...ice-library/src/servicelib/fastapi/dependencies.py 0.0% <0.0%> (ø)
.../service-library/src/servicelib/fastapi/openapi.py 90.2% <90.2%> (ø)
...ackages/service-library/src/servicelib/__init__.py 100.0% <100.0%> (ø)
.../service-library/src/servicelib/functools_utils.py 100.0% <100.0%> (ø)
...og/src/simcore_service_catalog/core/application.py 83.3% <100.0%> (+0.7%) ⬆️
...ce_webserver/resource_manager/garbage_collector.py 73.1% <0.0%> (-0.9%) ⬇️
.../director/src/simcore_service_director/producer.py 61.2% <0.0%> (-0.3%) ⬇️
... and 1 more

@pcrespov pcrespov added a:services-library issues on packages/service-libs t:maintenance Some planned maintenance work labels Sep 22, 2021
@pcrespov pcrespov changed the title WIP: data validation and fastapi tooling ✨♻️🐛 projects tables validation scripts and fastapi tooling Sep 22, 2021
@pcrespov pcrespov changed the title ✨♻️🐛 projects tables validation scripts and fastapi tooling ✨♻️🐛 serviceslib[fastapi] and projects tables validation scripts Sep 22, 2021
@pcrespov pcrespov marked this pull request as ready for review September 22, 2021 21:15
@pcrespov pcrespov requested review from sanderegg, GitHK, colinRawlings, mguidon and odeimaiz and removed request for sanderegg September 22, 2021 21:15
Copy link
Contributor

@GitHK GitHK left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cleanup looks very nice.

Below some minor stuff and possible concerns.

packages/service-library/Makefile Show resolved Hide resolved
packages/service-library/setup.py Outdated Show resolved Hide resolved
scripts/common.Makefile Show resolved Hide resolved
services/catalog/setup.py Outdated Show resolved Hide resolved
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looks very neat!
maybe check the issues with exclusiveMinimum that looks suspicious

services/catalog/openapi.json Show resolved Hide resolved
services/catalog/openapi.json Show resolved Hide resolved
Copy link
Member

@sanderegg sanderegg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

great!

@pcrespov pcrespov merged commit c815eb8 into ITISFoundation:master Sep 24, 2021
@pcrespov pcrespov deleted the is544/validate-projects branch September 24, 2021 13:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
a:services-library issues on packages/service-libs t:maintenance Some planned maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants