⬆️ Maintenance/test and tools [week 19] #3031

pcrespov · 2022-05-09T13:16:08Z

What do these changes do?

⬆️ Periodic update of tests and tool libraries .
⬆️ Fixed missing dependency downstream in dask-task-models-library: when jsonschema was introduced in models_library, we forgot to pip-compile downstream.
⬆️ Fixed storage tests requirements: Any change in _test.in (add/remove/modify) requires pip-compile it.
✨ Improved reporting scripts to scan for packages affected by the changes

Summary

This is a summary of the libraries updated in this PRs:

#packages before: 50
#packages after : 53

where 🔧=tools and 🧪= test requirements.

#	name	before	after	upgrade	count	packages
1	aio-pika	7.1.2	7.2.0	minor	1	swarm-deploy🧪
2	aioboto3	9.4.0, 9.5.0	9.6.0	minor	2	director-v2🧪 simcore-sdk🧪
3	aiobotocore	2.1.2, 2.2.0	2.3.0	minor	2	director-v2🧪 simcore-sdk🧪
4	astroid	2.11.2, 2.11.3	2.11.5		15	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🔧 models-library🧪 postgres-database🧪 service-integration🧪 service-library🧪 settings-library🧪 simcore-sdk🧪 storage🧪 web🧪
5	bcrypt	3.2.0	3.2.2		6	api-server🧪 catalog🧪 director-v2🧪 postgres-database🧪 service-library🧪 web🧪
6	black	22.1.0	22.3.0	minor	1	service-integration🔧
7	boto3	1.20.24	1.21.21	minor	1	director-v2🧪
8	botocore	1.23.24	1.24.21	minor	1	director-v2🧪
9	click	8.0.4, 8.1.2	8.1.3		16	dask-task-models-library🧪🔧 models-library🔧 postgres-database🧪🔧 public-api🧪 service-integration🧪🔧 service-library🔧 settings-library🧪🔧 simcore-sdk🧪🧪🔧 swarm-deploy🧪🧪
10	cryptography	36.0.2	37.0.2	MAJOR	5	catalog🧪 dask-sidecar🧪 director-v2🧪 postgres-database🧪 service-library🧪
11	dask	2022.4.0	2022.5.0	minor	1	dask-task-models-library🧪
12	dask-gateway-server	0.9.0	2022.4.0	MAJOR	1	director-v2🧪
13	distributed	2022.4.0	2022.5.0	minor	1	dask-task-models-library🧪
14	email-validator	1.1.3	1.2.1	minor	5	dask-task-models-library🧪 models-library🧪 service-integration🧪 simcore-sdk🧪 swarm-deploy🧪
15	faker	13.3.3, 13.3.4, 13.3.5	13.7.0	minor	14	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🧪 models-library🧪 postgres-database🧪 public-api🧪 service-library🧪 simcore-sdk🧪 storage🧪 web🧪
16	fastapi	0.75.1, 0.75.0	0.76.0	minor	2	helpers🧪 service-library🧪
17	hypothesis	6.39.6	6.46.2	minor	1	web🧪
18	icdiff	2.0.4	2.0.5		7	dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 models-library🧪 storage🧪 web🧪
19	identify	2.4.12	2.5.0	minor	17	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 simcore-sdk🔧 storage🔧 swarm-deploy🧪 web🔧
20	jinja2	3.1.1	3.1.2		1	dask-task-models-library🧪
21	jmespath	0.10.0	1.0.0	MAJOR	1	director-v2🧪
22	jsonschema	4.4.0	4.5.1	minor	4	models-library🧪 public-api🧪 service-integration🧪 tests🧪
23	locket	0.2.1	1.0.0	MAJOR	1	dask-task-models-library🧪
24	more-itertools	8.12.0	8.13.0	minor	1	tests🧪
25	mypy	0.942	0.950	minor	1	dynamic-sidecar🔧
26	pandas	1.4.1	1.4.2		1	storage🧪
27	paramiko	2.10.3	2.10.4		6	api-server🧪 catalog🧪 director-v2🧪 postgres-database🧪 service-library🧪 web🧪
28	pillow	9.0.1	9.1.0	minor	1	director-v2🧪
29	pint	0.19.1, 0.18	0.19.2		4	dask-task-models-library🧪 models-library🧪 simcore-sdk🧪 swarm-deploy🧪
30	pip-tools	6.5.1	6.6.0	minor	16	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 storage🔧 swarm-deploy🧪 web🔧
31	platformdirs	2.5.1	2.5.2		29	api-server🧪🔧 catalog🧪🔧 dask-sidecar🧪🔧 dask-task-models-library🧪🔧 datcore-adapter🧪🔧 director-v2🧪🔧 dynamic-sidecar🔧 models-library🧪🔧 postgres-database🧪🔧 public-api🧪 service-integration🧪🔧 service-library🧪🔧 settings-library🧪🔧 storage🧪🔧 swarm-deploy🧪 web🧪🔧
32	pre-commit	2.17.0, 2.18.1	2.19.0	minor	17	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 simcore-sdk🔧 storage🔧 swarm-deploy🧪 web🔧
33	ptvsd	4.3.2	🗑️ removed		1	storage🧪
34	pylint	2.13.2, 2.13.7, 2.13.4	2.13.8		15	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🔧 models-library🧪 postgres-database🧪 service-integration🧪 service-library🧪 settings-library🧪 simcore-sdk🧪 storage🧪 web🧪
35	pyparsing	3.0.7	3.0.8		14	dask-task-models-library🧪🧪 datcore-adapter🧪 environment-setup🧪 models-library🧪 postgres-database🧪 public-api🧪 service-integration🧪🧪 service-library🧪 settings-library🧪 storage🧪 swarm-deploy🧪 tests🧪
36	pytest	7.1.1	7.1.2		20	api-server🧪 catalog🧪 dask-sidecar🧪 dask-task-models-library🧪 datcore-adapter🧪 director-v2🧪 dynamic-sidecar🧪 environment-setup🧪 models-library🧪 postgres-database🧪 public-api🧪 service-integration🧪🧪 service-library🧪 settings-library🧪 simcore-sdk🧪 storage🧪 swarm-deploy🧪 tests🧪 web🧪
37	pytest-docker	0.11.0	0.12.0	minor	6	api-server🧪 catalog🧪 director-v2🧪 postgres-database🧪 service-library🧪 web🧪
38	sqlalchemy	1.4.32, 1.4.35	1.4.36		7	models-library🧪 postgres-database🧪🧪🧪 simcore-sdk🧪🧪 swarm-deploy🧪
39	starlette	0.17.1	0.18.0	minor	2	helpers🧪 service-library🧪
40	thrift	0.15.0	0.16.0	minor	1	service-library🧪
41	tqdm	4.63.1	4.64.0	minor	1	swarm-deploy🧪
42	types-aiofiles	0.8.5	0.8.8		1	dynamic-sidecar🧪
43	types-pyyaml	6.0.5	6.0.7		1	dynamic-sidecar🧪
44	typing-extensions	4.1.1	4.2.0	minor	23	dask-task-models-library🧪🧪🔧 environment-setup🧪 helpers🧪 models-library🧪🧪🔧 postgres-database🧪🔧 public-api🧪 service-integration🧪🧪🔧 service-library🧪🧪🧪🔧 settings-library🧪🧪🔧 swarm-deploy🧪🧪
45	virtualenv	20.14.0	20.14.1		16	api-server🔧 catalog🔧 dask-sidecar🔧 dask-task-models-library🔧 datcore-adapter🔧 director-v2🔧 dynamic-sidecar🔧 models-library🔧 postgres-database🔧 public-api🧪 service-integration🔧 service-library🔧 settings-library🔧 storage🔧 swarm-deploy🧪 web🔧
46	websocket-client	0.59.0, 1.3.1	1.3.2		3	public-api🧪 service-integration🧪 storage🧪
47	websockets	10.2	10.3	minor	1	web🧪
48	werkzeug	2.0.3	2.1.2	minor	1	tests🧪
49	wrapt	1.14.0	1.14.1		11	api-server🧪 catalog🧪 dask-task-models-library🧪 director-v2🧪 dynamic-sidecar🔧 models-library🧪 postgres-database🧪 service-integration🧪 service-library🧪 settings-library🧪 simcore-sdk🧪
50	zict	2.1.0	2.2.0	minor	1	dask-task-models-library🧪

Next steps

After some weeks without updates, I noticed that some of the libraries are used in different versions of the same dependency ( we denote this "dependency dispersion"). This will be corrected separately per service/library in following PRs.

Related issue/s

periodic maintenance of test and tooling libraries

How to test

Checklist

Add links to major changes
color affected packages
Openapi changes? make openapi-specs, git commit ... and then make version-*)
Database migration script? cd packages/postgres-database, make setup-commit, sc-pg review -m "my changes"
Unit tests for the changes exist
Runs in the swarm
Documentation reflects the changes
New module? Add your github username to .github/CODEOWNERS

codecov · 2022-05-09T13:18:12Z

Codecov Report

Merging #3031 (456f132) into master (25178f5) will increase coverage by 0.9%.
The diff coverage is n/a.

@@           Coverage Diff            @@
##           master   #3031     +/-   ##
========================================
+ Coverage    78.8%   79.7%   +0.9%     
========================================
  Files         698     698             
  Lines       29226   29226             
  Branches     3758    3758             
========================================
+ Hits        23043   23321    +278     
+ Misses       5357    5066    -291     
- Partials      826     839     +13

Flag	Coverage Δ
integrationtests	`66.1% <ø> (+1.4%)`	⬆️
unittests	`75.5% <ø> (+<0.1%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...ore_service_director_v2/utils/client_decorators.py	`73.3% <0.0%> (-3.4%)`	⬇️
...e_service_director_v2/modules/dask_clients_pool.py	`92.7% <0.0%> (-1.5%)`	⬇️
..._director_v2/modules/dynamic_sidecar/docker_api.py	`87.1% <0.0%> (-0.9%)`	⬇️
...simcore_service_director_v2/modules/dask_client.py	`91.9% <0.0%> (-0.7%)`	⬇️
.../director/src/simcore_service_director/producer.py	`62.7% <0.0%> (+0.8%)`	⬆️
.../simcore_service_catalog/services/access_rights.py	`82.5% <0.0%> (+1.2%)`	⬆️
...ector_v2/modules/dynamic_sidecar/scheduler/task.py	`83.2% <0.0%> (+1.8%)`	⬆️
...imcore_service_webserver/garbage_collector_core.py	`70.1% <0.0%> (+1.8%)`	⬆️
...core-sdk/src/simcore_sdk/node_data/data_manager.py	`94.1% <0.0%> (+1.9%)`	⬆️
...core-sdk/src/simcore_sdk/node_ports/_items_list.py	`95.9% <0.0%> (+4.0%)`	⬆️
... and 14 more

KZzizzle

as code owner, I approve this request

sanderegg

looking good. I'm a little bit worried about the mega update in dask-gateway. but let's see

GitHK

:+1 nice

sonarcloud · 2022-05-10T09:49:40Z

Kudos, SonarCloud Quality Gate passed!

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.6% Duplication

pcrespov added 5 commits May 9, 2022 15:06

testing folders

c1b238d

updates dask-taksmodel-library downstream and updates reqs

325c2dc

packages upgrade

3a74c97

services update only tools and tests

7aadc15

minor

c92bd14

pcrespov self-assigned this May 9, 2022

pcrespov changed the title ~~Maintenance/test and tools [week 19]~~ ⬆️ Maintenance/test and tools [week 19] May 9, 2022

pcrespov added t:maintenance Some planned maintenance work changelog:ignore-for-release labels May 9, 2022

pcrespov added this to the Croissant milestone May 9, 2022

pcrespov added 3 commits May 9, 2022 15:27

fixes reqs in service-lib

4342b55

ci

231e4a0

fixes storage reqs

c6c4306

pcrespov marked this pull request as ready for review May 9, 2022 15:55

pcrespov requested review from sanderegg, GitHK, mguidon and KZzizzle as code owners May 9, 2022 15:55

KZzizzle approved these changes May 9, 2022

View reviewed changes

pcrespov added 2 commits May 9, 2022 19:01

adds coverage

64511c9

changed to symbols since colors do not work

1a5e3cc

sanderegg approved these changes May 9, 2022

View reviewed changes

GitHK approved these changes May 10, 2022

View reviewed changes

Merge branch 'master' into maintenance/test-and-tools-week-19

456f132

pcrespov merged commit 375cbd3 into ITISFoundation:master May 10, 2022

pcrespov deleted the maintenance/test-and-tools-week-19 branch May 10, 2022 14:42

sanderegg mentioned this pull request Jun 2, 2022

maintenance/scaling of the platform ITISFoundation/osparc-issues#428

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

⬆️ Maintenance/test and tools [week 19] #3031

⬆️ Maintenance/test and tools [week 19] #3031

pcrespov commented May 9, 2022 •

edited

codecov bot commented May 9, 2022 •

edited

KZzizzle left a comment

sanderegg left a comment

GitHK left a comment

sonarcloud bot commented May 10, 2022

⬆️ Maintenance/test and tools [week 19] #3031

⬆️ Maintenance/test and tools [week 19] #3031

Conversation

pcrespov commented May 9, 2022 • edited

What do these changes do?

Summary

Next steps

Related issue/s

How to test

Checklist

codecov bot commented May 9, 2022 • edited

Codecov Report

KZzizzle left a comment

Choose a reason for hiding this comment

sanderegg left a comment

Choose a reason for hiding this comment

GitHK left a comment

Choose a reason for hiding this comment

sonarcloud bot commented May 10, 2022

pcrespov commented May 9, 2022 •

edited

codecov bot commented May 9, 2022 •

edited