Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

⬆️ Maintenance: upgrade dask libraries #3949

Merged

Conversation

sanderegg
Copy link
Member

@sanderegg sanderegg commented Mar 6, 2023

What do these changes do?

This brings a full upgrade and cleanup of dask-sidecar.
This brings dask, distributed and dask-gateway related upgrade to director-v2.

These upgrades might be subsequently upgraded again when more tests with the osparc-dask-gateway will be run. Also it will be necessary to bring the osparc-dask-gateway inside this repo to keep it in sync.

Highlights on updated libraries (only updated libraries are included)

  • #packages before: 105
  • #packages after : 58
# name before after upgrade count packages
1 aio-pika 8.2.5 9.0.4 MAJOR 1 dask-sidecar⬆️
2 aiobotocore 2.3.3 2.4.2 minor 1 dask-sidecar⬆️
3 aiofiles 0.8.0 23.1.0 MAJOR 1 dask-sidecar⬆️
4 aiohttp 3.8.3 3.8.4 2 dask-sidecar⬆️🧪
5 aioitertools 0.10.0 0.11.0 minor 1 dask-sidecar⬆️
6 aiormq 6.4.2 6.7.3 minor 1 dask-sidecar⬆️
7 aiosignal 1.2.0 1.3.1 minor 2 dask-sidecar⬆️🧪
8 anyio 3.6.1 🗑️ removed 1 dask-sidecar⬆️
9 argon2-cffi 20.1.0 🗑️ removed 1 dask-sidecar⬆️
10 astroid 2.14.2 2.15.0 minor 1 dask-sidecar🔧
11 async-generator 1.10 🗑️ removed 1 dask-sidecar⬆️
12 beautifulsoup4 4.10.0 🗑️ removed 1 dask-sidecar⬆️
13 black 22.12.0 23.1.0 MAJOR 1 dask-sidecar🔧
14 bleach 3.3.0 🗑️ removed 1 dask-sidecar⬆️
15 blosc 1.10.6 1.11.1 minor 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
16 bokeh 3.0.3, 2.4.3 🗑️ removed 2 dask-sidecar⬆️
director-v2🧪
17 botocore 1.24.21 1.27.59 minor 1 dask-sidecar⬆️
18 cffi 1.15.0 1.15.1 2 dask-sidecar⬆️🧪
19 charset-normalizer 2.0.12 3.1.0 MAJOR 2 dask-sidecar⬆️🧪
20 click 8.1.3 🗑️ removed 1 dask-sidecar⬆️
21 cloudpickle 2.0.0 2.2.1 minor 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
22 contourpy 1.0.7 🗑️ removed 1 director-v2🧪
23 cryptography 39.0.1 39.0.2 1 dask-sidecar🧪
24 cytoolz 0.11.0 🗑️ removed 1 dask-sidecar⬆️
25 dask 2022.9.2 2023.3.0 MAJOR 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
26 dask-gateway 2022.10.0 2023.1.1 MAJOR 2 dask-sidecar⬆️
director-v2⬆️
27 defusedxml 0.7.1 🗑️ removed 1 dask-sidecar⬆️
28 distributed 2022.9.2 2023.3.0 MAJOR 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
29 dnspython 2.0.0 2.3.0 minor 1 dask-sidecar⬆️
30 email-validator 1.2.1 1.3.1 minor 1 dask-sidecar⬆️
31 faker 17.4.0 17.6.0 minor 1 dask-sidecar🧪
32 fastjsonschema 2.15.3 🗑️ removed 1 dask-sidecar⬆️
33 frozenlist 1.3.0 1.3.3 2 dask-sidecar⬆️🧪
34 fsspec 2022.5.0 2023.3.0 MAJOR 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
35 heapdict 1.0.1 🗑️ removed 1 dask-sidecar⬆️
36 idna 2.10 3.4 MAJOR 2 dask-sidecar⬆️🧪
37 importlib-metadata 5.0.0 🗑️ removed 1 dask-sidecar⬆️
38 jinja2 3.1.2 🗑️ removed 2 dask-sidecar⬆️
director-v2🧪
39 jmespath 1.0.0 1.0.1 1 dask-sidecar⬆️
40 jupyter-client 6.1.12 🗑️ removed 1 dask-sidecar⬆️
41 jupyter-core 4.11.2 🗑️ removed 1 dask-sidecar⬆️
42 jupyter-server 1.18.1 🗑️ removed 1 dask-sidecar⬆️
43 jupyter-server-proxy 3.2.1 🗑️ removed 1 dask-sidecar⬆️
44 jupyterlab-pygments 0.1.2 🗑️ removed 1 dask-sidecar⬆️
45 locket 1.0.0 🗑️ removed 1 dask-sidecar⬆️
46 lz4 4.0.0 4.3.2 minor 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
47 markupsafe 2.1.1 2.1.2 5 dask-sidecar⬆️⬆️⬆️
director-v2⬆️🧪
48 mistune 2.0.4 🗑️ removed 1 dask-sidecar⬆️
49 msgpack 1.0.3 1.0.5 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
50 multidict 6.0.2 6.0.4 2 dask-sidecar⬆️🧪
51 nbclient 0.5.3 🗑️ removed 1 dask-sidecar⬆️
52 nbconvert 7.2.1 🗑️ removed 1 dask-sidecar⬆️
53 nbformat 5.3.0 🗑️ removed 1 dask-sidecar⬆️
54 nest-asyncio 1.5.1 🗑️ removed 1 dask-sidecar⬆️
55 numpy 1.22.3, 1.24.2 1.24.2 3 dask-sidecar⬆️⬆️
director-v2🧪
56 packaging 21.3 23.0 MAJOR 8 dask-sidecar⬆️⬆️⬆️🧪🔧
director-v2⬆️🧪🔧
57 pandas 1.2.4, 1.5.3 🗑️ removed 3 dask-sidecar⬆️⬆️
director-v2🧪
58 pandocfilters 1.4.3 🗑️ removed 1 dask-sidecar⬆️
59 partd 1.2.0 1.3.0 minor 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
60 pillow 9.3.0, 9.4.0 9.4.0 3 dask-sidecar⬆️⬆️
director-v2🧪
61 pip-tools 6.12.2 6.12.3 1 dask-sidecar🔧
62 platformdirs 3.0.0 3.1.0 minor 1 dask-sidecar🔧
63 prometheus-client 0.14.1 🗑️ removed 1 dask-sidecar⬆️
64 psutil 5.9.1 5.9.4 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
65 ptyprocess 0.7.0 🗑️ removed 1 dask-sidecar⬆️
66 pycparser 2.20 2.21 minor 2 dask-sidecar⬆️🧪
67 pydantic 1.9.0 1.10.2 minor 1 dask-sidecar⬆️
68 pygments 2.9.0 🗑️ removed 1 dask-sidecar⬆️
69 pyinstrument 4.3.0 4.4.0 minor 1 dask-sidecar⬆️
70 pylint 2.16.2 2.16.4 1 dask-sidecar🔧
71 pyparsing 3.0.9 🗑️ removed 8 dask-sidecar⬆️⬆️⬆️🧪🔧
director-v2⬆️🧪🔧
72 pyrsistent 0.18.1 0.19.3 minor 1 dask-sidecar⬆️
73 pytest 7.2.1 7.2.2 1 dask-sidecar🧪
74 python-dateutil 2.8.1 2.8.2 3 dask-sidecar⬆️⬆️🧪
75 python-dotenv 0.20.0 1.0.0 MAJOR 2 dask-sidecar⬆️🧪
76 pytz 2020.1, 2022.7.1 🗑️ removed 3 dask-sidecar⬆️⬆️
director-v2🧪
77 pyyaml 5.4.1 🗑️ removed 2 dask-sidecar⬆️
director-v2🧪
78 pyzmq 22.1.0 🗑️ removed 1 dask-sidecar⬆️
79 redis 4.4.0 4.5.1 minor 1 dask-sidecar⬆️
80 requests 2.27.1 2.28.2 minor 2 dask-sidecar⬆️🧪
81 s3fs 2022.5.0 2023.3.0 MAJOR 1 dask-sidecar⬆️
82 send2trash 1.7.1 🗑️ removed 1 dask-sidecar⬆️
83 simpervisor 0.4 🗑️ removed 1 dask-sidecar⬆️
84 six 1.15.0 1.16.0 minor 3 dask-sidecar⬆️⬆️🧪
85 sniffio 1.2.0 🗑️ removed 1 dask-sidecar⬆️
86 sortedcontainers 2.4.0 🗑️ removed 1 dask-sidecar⬆️
87 soupsieve 2.3.2 🗑️ removed 1 dask-sidecar⬆️
88 tblib 1.7.0 🗑️ removed 1 dask-sidecar⬆️
89 tenacity 8.1.0 8.2.2 minor 1 dask-sidecar⬆️
90 terminado 0.10.1 🗑️ removed 1 dask-sidecar⬆️
91 tinycss2 1.1.1 🗑️ removed 1 dask-sidecar⬆️
92 toolz 0.11.1 0.12.0 minor 4 dask-sidecar⬆️⬆️⬆️
director-v2⬆️
93 tornado 6.1 6.2 minor 5 dask-sidecar⬆️⬆️⬆️
director-v2⬆️🧪
94 tqdm 4.64.1 4.65.0 minor 1 dask-sidecar⬆️
95 traitlets 5.1.1 🗑️ removed 1 dask-sidecar⬆️
96 typer 0.6.1 0.7.0 minor 1 dask-sidecar⬆️
97 typing-extensions 4.3.0 4.5.0 minor 3 dask-sidecar⬆️⬆️🔧
98 urllib3 1.26.9 1.26.14 6 dask-sidecar⬆️⬆️⬆️🧪
director-v2⬆️🧪
99 webencodings 0.5.1 🗑️ removed 1 dask-sidecar⬆️
100 websocket-client 0.59.0 1.5.1 MAJOR 2 dask-sidecar⬆️🧪
101 wrapt 1.14.1 1.15.0 minor 2 dask-sidecar⬆️🔧
102 xyzservices 2023.2.0 🗑️ removed 1 director-v2🧪
103 yarl 1.7.2 1.8.2 minor 2 dask-sidecar⬆️🧪
104 zict 2.2.0 🗑️ removed 1 dask-sidecar⬆️
105 zipp 3.9.0 🗑️ removed 1 dask-sidecar⬆️

Legend:

  • ⬆️ base dependency (only services because packages are floating)
  • 🧪 test dependency
  • 🔧 tool dependency

Related issue/s

fixes ITISFoundation/osparc-issues#885
fixes ITISFoundation/osparc-issues#675

How to test

Checklist

@sanderegg sanderegg added the t:maintenance Some planned maintenance work label Mar 6, 2023
@sanderegg sanderegg added this to the Mithril milestone Mar 6, 2023
@sanderegg sanderegg self-assigned this Mar 6, 2023
@codecov
Copy link

codecov bot commented Mar 6, 2023

Codecov Report

Merging #3949 (2253446) into master (7d51574) will decrease coverage by 4.1%.
The diff coverage is 100.0%.

Impacted file tree graph

@@           Coverage Diff            @@
##           master   #3949     +/-   ##
========================================
- Coverage    86.7%   82.7%   -4.1%     
========================================
  Files         556     699    +143     
  Lines       27154   31771   +4617     
  Branches      438     143    -295     
========================================
+ Hits        23568   26290   +2722     
- Misses       3456    5433   +1977     
+ Partials      130      48     -82     
Flag Coverage Δ
integrationtests 66.6% <100.0%> (+5.0%) ⬆️
unittests 78.8% <100.0%> (-5.0%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
...src/dask_task_models_library/container_tasks/io.py 100.0% <ø> (ø)
...simcore_service_director_v2/modules/dask_client.py 92.8% <ø> (ø)
...dask_sidecar/computational_sidecar/docker_utils.py 90.8% <100.0%> (ø)
...car/src/simcore_service_dask_sidecar/dask_utils.py 96.7% <100.0%> (+<0.1%) ⬆️
...ore_service_director_v2/models/schemas/clusters.py 98.8% <100.0%> (+<0.1%) ⬆️
...mic_sidecar/docker_service_specs/volume_remover.py 100.0% <100.0%> (ø)
...rc/simcore_service_webserver/version_control_db.py 24.5% <0.0%> (-68.7%) ⬇️
...simcore_service_webserver/meta_modeling_results.py 30.9% <0.0%> (-59.6%) ⬇️
...service_webserver/meta_modeling_version_control.py 27.1% <0.0%> (-57.2%) ⬇️
...core_service_webserver/meta_modeling_iterations.py 31.2% <0.0%> (-56.5%) ⬇️
... and 385 more

@sanderegg sanderegg force-pushed the maintenance/upgrade_dask_libraries branch from 32749dc to a90383c Compare March 7, 2023 13:55
@sanderegg sanderegg force-pushed the maintenance/upgrade_dask_libraries branch from 3fae0cf to 270800d Compare March 7, 2023 16:02
@sanderegg sanderegg marked this pull request as ready for review March 8, 2023 22:27
Copy link
Member

@pcrespov pcrespov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

excellent!

@sonarcloud
Copy link

sonarcloud bot commented Mar 9, 2023

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

@codeclimate
Copy link

codeclimate bot commented Mar 9, 2023

Code Climate has analyzed commit 2253446 and detected 0 issues on this pull request.

View more on Code Climate.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
t:maintenance Some planned maintenance work
Projects
None yet
Development

Successfully merging this pull request may close these issues.

AppTeam Std Simulations on S4L/AWS M1-12 Maintenance and DevOps
4 participants