⬆️ Maintenance: fixes downstream upgrades and reduces version dispersion

[pcrespov]

What do these changes do?

This PR address requirement updates across the entire repository.

Sometimes, after changing requirements in packages we forget to update downstream. This creates inconsistencies when installing services between the frozen lists (_base.txt) and the installed libraries (see make install-ci pip-freeze).

Another issue is the different versions of any given library listed in different requirements across the repository. We denote this as "version dispersion" and aim to reduce it as much as the constraints set allow. For that we use selective updates of libraries based on the second table overview below.

Specifics:

• 🐛 Fixes missing downstream upgrades (i.e. servicelib reqs were updated forgot to update downstream)
• ⬆️ batch upgrade of libraries to reduce version dispersion. We used a script like:

packages=orjson,packaging,prometheus-client, # ... and many more

for u in ${packages//,/ }
do 
   make reqs-all upgrade=$u &> reqs-$u.log
   git commit -am "upgrades $u" --no-verify
done

• Critical libraries (e.g. fastapi, boto3 ...) were not included in this update
• 📝 See details in requirements/how-to-unify-versions.md
• See commits/tables below to see how and which library was updated

Highlights on updated libraries (only updated libraries are included)

• #packages before: 42
• #packages after : 39

#	name	before	after	upgrade	count	packages
1	alembic	1.7.4, 1.7.7, 1.7.5	1.8.0	minor	16	api-server⬆️🧪 catalog⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️ models-library🧪 postgres-database🧪🧪 simcore-sdk🧪🧪 storage⬆️ swarm-deploy🧪 web⬆️🧪
2	anyio	3.2.1, 3.3.4, 3.2.0, 3.5.0	3.6.1	minor	10	api-server⬆️🧪 catalog⬆️🧪 dask-sidecar⬆️ director-v2⬆️🧪 dynamic-sidecar⬆️ public-api🧪 service-library🧪
3	asgiref	3.4.1, 3.5.0	3.5.2		4	api-server⬆️ catalog⬆️ director-v2⬆️ dynamic-sidecar⬆️
4	async-timeout	4.0.1	4.0.2		2	dask-sidecar⬆️🧪
5	attrs	21.4.0, 20.2.0	20.3.0	🔥 downgrade	4	catalog🧪 dask-sidecar⬆️🧪 datcore-adapter🧪
6	certifi	2020.12.5, 2021.10.8, 2021.5.30	2022.5.18.1	MAJOR	25	api-server⬆️🧪 catalog⬆️🧪 dask-sidecar⬆️🧪 dask-task-models-library🧪 director-v2⬆️🧪 dynamic-sidecar⬆️🧪 e2e🧪 models-library🧪 postgres-database🧪🧪 public-api🧪 service-integration🧪🧪 service-library🧪 settings-library🧪 simcore-sdk🧪 storage⬆️🧪 swarm-deploy🧪 web🧪
7	cffi	1.14.5	1.15.0	minor	3	dask-sidecar⬆️🧪 dynamic-sidecar⬆️
8	chardet	4.0.0, 3.0.4	🗑️ removed		6	catalog⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️🧪
9	charset-normalizer	2.0.7, 2.0.10, 2.0.6	2.0.12		6	catalog🧪 dask-sidecar⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️
10	click	8.1.2, 8.0.3, 8.0.4	8.1.3		14	api-server⬆️🧪🔧 dask-sidecar⬆️⬆️⬆️⬆️🔧 director-v2⬆️🔧 dynamic-sidecar⬆️🔧 storage⬆️🔧
11	docker	5.0.2	5.0.3		1	dynamic-sidecar⬆️
12	email-validator	1.1.3, 1.1.1, 1.1.2	1.2.1	minor	8	api-server⬆️ catalog⬆️ dask-sidecar⬆️⬆️ datcore-adapter⬆️ director-v2⬆️ dynamic-sidecar⬆️ storage⬆️
13	frozenlist	1.2.0	1.3.0	minor	2	dask-sidecar⬆️🧪
14	jinja2	3.1.1, 2.11.3	3.1.2		5	api-server⬆️🧪🔧 catalog⬆️ storage⬆️
15	jsonschema	4.5.1	4.6.0,3.2.0	minor	5	dask-task-models-library🧪 models-library🧪 public-api🧪 service-integration🧪 tests🧪
16	markupsafe	1.1.1, 2.0.1	2.1.1	minor	9	catalog⬆️🧪 dask-sidecar⬆️⬆️⬆️⬆️ director-v2⬆️🧪 dynamic-sidecar⬆️
17	multidict	5.1.0, 5.2.0	6.0.2	MAJOR	8	catalog⬆️🧪 dask-sidecar⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️🧪
18	orjson	3.5.4, 3.4.8, 3.6.8, 3.6.7	3.7.2	minor	4	api-server⬆️ catalog⬆️ director-v2⬆️ web⬆️
19	packaging	20.4, 21.0, 20.9	21.3	minor	11	catalog⬆️🧪 dask-sidecar⬆️⬆️⬆️⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️🧪
20	paramiko	2.10.3, 2.10.4	2.11.0	minor	2	catalog🧪 dynamic-sidecar⬆️
21	pint	0.19.1	0.19.2		3	api-server⬆️ director-v2⬆️ dynamic-sidecar⬆️
22	prometheus-client	0.13.1, 0.11.0	0.14.1	minor	3	dask-sidecar⬆️ service-library🧪 storage⬆️
23	psutil	5.9.0, 5.8.0	5.9.1		6	dask-sidecar⬆️⬆️⬆️⬆️ dask-task-models-library🧪 director-v2⬆️
24	psycopg2-binary	2.9.2, 2.9.1, 2.8.6	2.9.3		5	catalog⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️
25	pyinstrument	3.4.2, 4.0.3	4.1.1	minor	4	catalog⬆️ dask-sidecar⬆️ director-v2⬆️ dynamic-sidecar⬆️
26	pyinstrument-cext	0.2.4	🗑️ removed		1	dask-sidecar⬆️
27	pyparsing	3.0.7, 3.0.8, 2.4.7	3.0.9		29	api-server⬆️🧪 catalog⬆️🧪 dask-sidecar⬆️⬆️⬆️⬆️🧪 dask-task-models-library🧪🧪 datcore-adapter🧪 director-v2⬆️🧪 dynamic-sidecar⬆️🧪 environment-setup🧪 models-library🧪 postgres-database🧪 public-api🧪 service-integration🧪🧪 service-library🧪 settings-library🧪 simcore-sdk🧪🧪 storage🧪 swarm-deploy🧪 tests🧪
28	pyrsistent	0.17.3, 0.18.0	0.18.1		2	dask-sidecar⬆️ dynamic-sidecar⬆️
29	python-dotenv	0.18.0, 0.19.0, 0.15.0	0.20.0	minor	7	catalog⬆️🧪 dask-sidecar⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️
30	requests	2.25.1, 2.26.0	2.27.1	minor	8	catalog⬆️🧪 dask-sidecar⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️🧪
31	six	1.15.0	🗑️ removed		1	dask-sidecar⬆️
32	sqlalchemy	1.4.31, 1.4.32, 1.4.36	1.4.37		18	api-server⬆️🧪 catalog⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️ models-library🧪 postgres-database🧪🧪🧪 service-library🧪 simcore-sdk🧪🧪 storage⬆️ swarm-deploy🧪 web⬆️🧪
33	tenacity	6.3.1, 6.2.0, 7.0.0	8.0.1	MAJOR	3	catalog⬆️ dask-sidecar⬆️ director-v2⬆️
34	thrift	0.13.0, 0.15.0	0.16.0	minor	4	api-server⬆️ catalog⬆️ director-v2⬆️ dynamic-sidecar⬆️
35	typing-extensions	4.1.1, 3.10.0.2	4.2.0	minor	14	api-server⬆️🧪🔧 catalog⬆️🧪🔧 director-v2⬆️🧪🔧 dynamic-sidecar⬆️🔧 storage⬆️🧪🔧
36	ujson	5.1.0, 4.0.2, 5.2.0	5.3.0,4.3.0	minor	4	catalog⬆️ director-v2⬆️ storage⬆️ web⬆️
37	urllib3	1.26.7, 1.26.6	1.26.9		11	catalog⬆️🧪 dask-sidecar⬆️⬆️⬆️⬆️🧪 director-v2⬆️🧪 dynamic-sidecar⬆️🧪
38	uvloop	0.15.2, 0.14.0	0.16.0	minor	2	catalog⬆️ director-v2⬆️
39	watchgod	0.7, 0.8.1, 0.6	0.8.2		3	api-server⬆️ catalog⬆️ director-v2⬆️
40	wrapt	1.14.0	1.14.1		4	dask-sidecar⬆️🧪 storage⬆️🧪
41	yarl	1.6.3	1.7.2	minor	5	catalog⬆️🧪 dask-sidecar⬆️🧪 dynamic-sidecar⬆️
42	zict	2.0.0	2.2.0	minor	5	dask-sidecar⬆️⬆️⬆️⬆️ director-v2⬆️

Legend:

• ⬆️ base dependency (only services because packages are floating)
• 🧪 test dependency
• 🔧 tool dependency

Repo-wide overview of libraries

• #reqs files parsed: 57

This table shows the version dispersion across the repository after this PR

#	name	versions-base	versions-test	versions-tool
1	aio-pika	6.8.0, 7.2.0	6.8.0, 7.2.0
2	aioboto3		9.6.0
3	aiobotocore	2.2.0	2.3.0
4	aiocache	0.11.1	0.11.1
5	aiodebug	2.3.0	2.3.0
6	aiodocker	0.19.1, 0.21.0	0.21.0
7	aiofiles	0.8.0	0.8.0
8	aiohttp	3.8.1	3.8.1
9	aiohttp-jinja2	1.5
10	aiohttp-security	0.4.0
11	aiohttp-session	2.11.0
12	aiohttp-swagger	1.0.16
13	aioitertools	0.10.0	0.10.0
14	aiopg	1.3.3	1.3.3
15	aioredis	2.0.1
16	aioresponses		0.7.3
17	aiormq	3.3.1, 6.2.3	3.3.1, 6.2.3
18	aiosignal	1.2.0	1.2.0
19	aiosmtplib	1.1.6
20	aiozipkin	1.1.1
21	alembic	1.8.0	1.8.0
22	anyio	3.5.0, 3.6.1	3.5.0, 3.6.1
23	argon2-cffi	20.1.0
24	asgi-lifespan		1.0.1
25	asgiref	3.5.0, 3.5.2
26	astroid		2.11.5	2.11.5
27	async-asgi-testclient		1.4.10
28	async-generator	1.10
29	async-timeout	4.0.2	4.0.2
30	asyncpg	0.25.0
31	attrs	20.3.0, 21.4.0	20.3.0, 21.4.0
32	aws-sam-translator		1.45.0
33	aws-xray-sdk		2.9.0
34	bcrypt	3.2.0	3.2.2
35	beautifulsoup4	4.10.0
36	black			22.3.0
37	bleach	3.3.0
38	blosc	1.10.6
39	bokeh	2.4.2	2.4.2
40	boto3	1.21.33	1.21.21, 1.23.4
41	boto3-stubs		1.23.9
42	botocore	1.24.21, 1.24.33	1.24.21, 1.26.4
43	botocore-stubs		1.26.9
44	bump2version			1.0.1
45	certifi	2021.10.8, 2022.5.18.1	2021.10.8, 2022.5.18.1
46	cffi	1.15.0	1.15.0
47	cfgv			3.3.1
48	cfn-lint		0.60.1
49	change-case			0.5.2
50	charset-normalizer	2.0.12	2.0.12
51	click	8.0.3, 8.1.2, 8.1.3	8.0.3, 8.1.3	8.0.3, 8.1.2, 8.1.3
52	cloudpickle	2.0.0
53	codecov		2.1.12
54	colorlog		6.6.0
55	configparser	5.2.0
56	coverage		6.3.2, 6.4
57	coveralls		3.3.1
58	cryptography	3.4.7, 36.0.2, 37.0.2	36.0.2, 37.0.2
59	cytoolz	0.11.0
60	dask	2022.4.0, 2022.5.0
61	dask-gateway	0.9.0
62	dask-gateway-server		2022.4.0
63	decorator	4.4.2
64	defusedxml	0.7.1
65	deprecated	1.2.13	1.2.13
66	dill		0.3.4, 0.3.5	0.3.5.1
67	distlib			0.3.4
68	distributed	2022.4.0, 2022.5.0
69	distro	1.5.0	1.7.0
70	dnspython	2.0.0, 2.1.0, 2.2.1	2.2.1
71	docker	5.0.3	5.0.3
72	docker-compose	1.29.1	1.29.1
73	dockerpty	0.4.1	0.4.1
74	docopt	0.6.2	0.6.2
75	ecdsa	0.14.1	0.17.0
76	email-validator	1.2.1	1.2.1
77	entrypoints	0.3
78	et-xmlfile	1.1.0
79	execnet		1.9.0
80	expiringdict	1.2.1
81	faker		13.7.0, 13.11.0, 13.11.1
82	fastapi	0.71.0, 0.75.0, 0.75.1
83	fastapi-contrib	0.2.11
84	fastapi-pagination	0.9.1
85	fastjsonschema	2.15.3
86	filelock			3.7.0
87	flaky		3.7.0
88	flask		2.1.2
89	flask-cors		3.0.10
90	frozenlist	1.3.0	1.3.0
91	fsspec	2022.3.0
92	future	0.18.2	0.18.2
93	futures	3.0.5
94	graphql-core		3.2.1
95	greenlet	1.1.2	1.1.2
96	gunicorn	20.1.0
97	h11	0.12.0	0.12.0
98	h2	4.1.0
99	heapdict	1.0.1
100	hpack	4.0.0
101	httpcore	0.15.0	0.15.0
102	httptools	0.2.0, 0.4.0
103	httpx	0.23.0	0.23.0
104	hyperframe	6.0.1
105	hypothesis		6.46.3
106	icdiff		2.0.5
107	identify			2.5.1
108	idna	2.10, 3.3	2.10, 3.3
109	importlib-metadata		4.11.3
110	iniconfig	1.1.1	1.1.1
111	inotify			0.2.10
112	isodate	0.6.1
113	isort		5.10.1	5.10.1
114	itsdangerous	1.1.0, 2.1.2	2.1.2
115	jaeger-client	4.8.0
116	jinja-app-loader	1.0.2
117	jinja2	2.11.3, 3.1.2	2.11.3, 3.1.2	3.1.2
118	jmespath	1.0.0	1.0.0
119	jschema-to-python		1.2.3
120	json2html	1.3.0
121	jsondiff	2.0.0	2.0.0
122	jsonpatch		1.32
123	jsonpickle		2.2.0
124	jsonpointer		2.3
125	jsonschema	3.2.0, 4.6.0	3.2.0, 4.6.0
126	junit-xml		1.9
127	jupyter-client	6.1.12
128	jupyter-core	4.7.1
129	jupyter-server	1.16.0
130	jupyter-server-proxy	3.2.1
131	jupyterlab-pygments	0.1.2
132	lazy-object-proxy	1.4.3	1.4.3, 1.7.1	1.7.1
133	locket	0.2.1, 1.0.0
134	lz4	4.0.0
135	mako	1.1.5, 1.2.0	1.1.5, 1.2.0
136	markupsafe	2.1.1	2.1.1	2.1.1
137	mccabe		0.7.0	0.7.0
138	minio	7.0.4	7.0.4
139	mistune	0.8.4
140	moto		3.1.9
141	msgpack	1.0.3
142	multidict	6.0.2	6.0.2
143	mypy			0.960
144	mypy-extensions			0.4.3
145	nbclient	0.5.3
146	nbconvert	6.4.5
147	nbformat	5.3.0
148	nest-asyncio	1.5.1
149	networkx	2.5.1	2.8.1
150	nodeenv			1.6.0
151	nose			1.3.7
152	numpy	1.22.3	1.22.3
153	openapi-core	0.12.0
154	openapi-schema-validator	0.2.3	0.2.3
155	openapi-spec-validator	0.4.0	0.4.0
156	openpyxl	3.0.9
157	opentracing	2.4.0
158	orjson	3.7.2
159	packaging	21.3	21.3
160	pamqp	2.3.0, 3.1.0	2.3.0, 3.1.0
161	pandas	1.2.4	1.4.2
162	pandocfilters	1.4.3
163	paramiko	2.11.0	2.10.4, 2.11.0
164	parfive	1.5.1
165	partd	1.2.0
166	passlib	1.7.4	1.7.4
167	pathspec			0.9.0
168	pbr		5.9.0
169	pennsieve	6.1.2
170	pep517			0.12.0
171	pillow	9.0.1	9.1.0
172	pint	0.19.2	0.19.2
173	pip-tools			6.6.2
174	platformdirs		2.5.2	2.5.2
175	pluggy	1.0.0	1.0.0
176	pprintpp		0.4.0
177	pre-commit			2.19.0
178	prometheus-client	0.14.1
179	protobuf	3.20.0
180	psutil	5.9.0, 5.9.1
181	psycopg2-binary	2.9.3	2.9.3
182	ptvsd		4.3.2
183	ptyprocess	0.7.0
184	py	1.11.0	1.11.0
185	py-cpuinfo		8.0.0
186	pyasn1	0.4.8	0.4.8
187	pycparser	2.20, 2.21	2.20, 2.21
188	pydantic	1.9.0	1.9.0
189	pyftpdlib		1.5.6
190	pygments	2.9.0
191	pyinstrument	3.4.2, 4.1.1	4.1.1
192	pyinstrument-cext	0.2.4
193	pylint		2.13.8, 2.13.9	2.13.9
194	pynacl	1.4.0	1.5.0
195	pyopenssl		22.0.0
196	pyparsing	3.0.9	3.0.9
197	pyrsistent	0.18.1	0.18.1
198	pytest	7.1.2	7.1.2
199	pytest-aiohttp		1.0.4
200	pytest-asyncio		0.18.3
201	pytest-benchmark		3.4.1
202	pytest-cov		3.0.0
203	pytest-docker		0.12.0
204	pytest-forked		1.4.0
205	pytest-icdiff		0.5
206	pytest-instafail		0.4.2
207	pytest-lazy-fixture		0.6.3
208	pytest-localftpserver		1.1.3
209	pytest-mock		3.7.0
210	pytest-runner		6.0.0
211	pytest-sugar		0.9.4
212	pytest-xdist		2.5.0
213	python-dateutil	2.8.1, 2.8.2	2.8.1, 2.8.2
214	python-dotenv	0.20.0	0.20.0
215	python-engineio	3.14.2
216	python-jose	3.2.0	3.3.0
217	python-magic	0.4.25
218	python-multipart	0.0.5
219	python-socketio	4.6.1
220	pytz	2020.1, 2022.1	2022.1
221	pyyaml	5.4.1, 6.0	5.4.1, 6.0	5.4.1, 6.0
222	pyzmq	22.1.0
223	redis	4.3.1	4.3.1
224	requests	2.27.1	2.27.1
225	responses		0.20.0
226	respx		0.19.2
227	rfc3986	1.4.0, 1.5.0	1.4.0, 1.5.0
228	rsa	4.0	4.8
229	s3fs	2022.3.0
230	s3transfer	0.5.2	0.5.2
231	sarif-om		1.0.4
232	semantic-version	2.9.0
233	semver	2.13.0
234	send2trash	1.7.1
235	setproctitle	1.2.3
236	simpervisor	0.4
237	six	1.15.0, 1.16.0	1.15.0, 1.16.0	1.15.0, 1.16.0
238	sniffio	1.2.0	1.2.0
239	sortedcontainers	2.4.0	2.4.0
240	soupsieve	2.3.2
241	sqlalchemy	1.4.37	1.4.37
242	sshpubkeys		3.3.1
243	starlette	0.17.1
244	strict-rfc3339	0.7
245	tblib	1.7.0
246	tenacity	8.0.1	8.0.1
247	termcolor		1.1.0
248	terminado	0.10.1
249	testpath	0.5.0
250	texttable	1.6.3	1.6.4
251	threadloop	1.0.2
252	thrift	0.16.0
253	toml			0.10.2
254	tomli	2.0.1	2.0.1	2.0.1
255	toolz	0.11.1, 0.11.2
256	tornado	6.1	6.1
257	tqdm	4.62.3, 4.63.1, 4.64.0	4.64.0
258	traitlets	5.1.1	5.1.1
259	typer	0.4.1	0.4.1	0.4.1
260	types-aiofiles		0.8.8
261	types-boto3		1.0.2
262	types-pkg-resources		0.1.3
263	types-pyyaml		6.0.7
264	typing-extensions	4.1.1, 4.2.0	4.1.1, 4.2.0	4.1.1, 4.2.0
265	ujson	4.3.0, 5.3.0
266	urllib3	1.26.9	1.26.9
267	uvicorn	0.15.0, 0.17.0, 0.17.6
268	uvloop	0.16.0
269	virtualenv			20.14.1
270	watchdog	2.1.5		2.1.8
271	watchgod	0.8.2
272	webencodings	0.5.1
273	websocket-client	0.59.0, 1.3.2	0.59.0, 1.3.2
274	websockets	10.1, 10.2	10.3
275	werkzeug	2.0.3, 2.1.2	2.1.2
276	wheel			0.37.1
277	wrapt	1.14.0, 1.14.1	1.14.0, 1.14.1	1.14.1
278	xmltodict		0.13.0
279	yarl	1.5.1, 1.7.2	1.5.1, 1.7.2
280	zict	2.2.0
281	zipp		3.8.0

[codecov]

Codecov Report

Merging #3092 (8650ee2) into master (3c24b83) will increase coverage by 0.0%.
The diff coverage is n/a.

@@          Coverage Diff           @@
##           master   #3092   +/-   ##
======================================
  Coverage    80.9%   81.0%           
======================================
  Files         716     716           
  Lines       30924   30924           
  Branches     4035    4035           
======================================
+ Hits        25042   25052   +10     
+ Misses       5006    4998    -8     
+ Partials      876     874    -2     

Flag	Coverage Δ
integrationtests	66.2% <ø> (-0.1%)	⬇️
unittests	76.9% <ø> (+<0.1%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
...imcore_service_catalog/db/repositories/services.py	70.0% <0.0%> (-0.7%)	⬇️
..._director_v2/modules/dynamic_sidecar/docker_api.py	87.2% <0.0%> (+0.4%)	⬆️
.../director/src/simcore_service_director/producer.py	62.1% <0.0%> (+0.4%)	⬆️
...src/simcore_service_webserver/socketio/handlers.py	82.3% <0.0%> (+0.9%)	⬆️
...ce_webserver/resource_manager/websocket_manager.py	94.8% <0.0%> (+1.2%)	⬆️
.../simcore_service_catalog/services/access_rights.py	81.2% <0.0%> (+2.5%)	⬆️
...ector_v2/modules/comp_scheduler/background_task.py	91.6% <0.0%> (+8.3%)	⬆️

[GitHK]

Nice additions!

Please see my comments below.

[KZzizzle]

👍

[sanderegg]

I see we have 2 different versions of dask/distributed. I think that is not good as they usually do not like having different versions, do we know why?
also, how do you define what is a critical library (you wrote that in the PR description)? Why aioboto3? or fastapi?

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.5% Duplication

[pcrespov]

I see we have 2 different versions of dask/distributed. I think that is not good as they usually do not like having different versions, do we know why? also, how do you define what is a critical library (you wrote that in the PR description)? Why aioboto3? or fastapi?

@sanderegg this is a bulk upgrade where I try to minimize the risk of incompatibility. I call critical libraries those that we mainly use directly in the development of some of the services/libraries in osparc-simcore. E.g. we use a lot of the functions in fastapi, aiohttp etc directly in our code. For those I prefer to have separate PRs since experience show that upgrades of those have a higher risk of breaking our code.

In the case of aioboto3 I did not want to upgrade until you merge your changes in storage

For dask/distributed we will do a separate upgrade (notice that the difference is in the package not in the service requirements). Moreover, we can enforce that those always have the same version