Secure Dask backend using self-signed certificates (⚠️ devops)

[sanderegg]

What do these changes do?

This PR basically changes the protocol used by the whole dask backend to enable TLS security (see https://distributed.dask.org/en/latest/tls.html) and also encrypts communication between all the dask parts (client, scheduler, workers). Notable exception to this rule is the dask-scheduler dashboard that is still in unprotected, but is also read-only so deemed as a lesser issue.

To this end the Makefile was adapted to ensure a self-signed certificate is created before the stack may be deployed locally (see make certificates in services/dask-sidecar). These certificates are git-ignored and are not deleted unless make clean is run. The certificates are passed as docker secrets inside the different dask-aware services (dask-sidecar, dask-scheduler, director-v2, clusters-keeper and computational-autoscaling).

Currently the certificates are the same for all the osparc-defined clusters (default cluster and external clusters that are created by clusters-keeper) for sanity.

details

• improve pytest-simcore docker_stack error messages to become more dev-friendly
• improved tests in clusters-keeper to find out when ENVs are missing to run the docker-compose file

Related issue/s

• fixes Secure connection to external computational clusters #4727

How to test

Dev Checklist

• No ENV changes or I properly updated ENV (read the instruction)

DevOps Checklist

osparc-ops-deployment-configuration MRs:

• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/327
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/331
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/332
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/333
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/334
• https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/336

osparc-ops-environments PRs:

• Comp-backend: Ensure dask secrets are relative to stack.yml file osparc-ops-environments#536
• fix typo osparc-ops-environments#538

BREAKING CHANGES MRs:

• staging: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/337
• production: https://git.speag.com/oSparc/osparc-ops-deployment-configuration/-/merge_requests/338

details

1. the dask-certificates are made available via docker secrets to the dask-aware services (i.e. dask-sidecar/scheduler, director-v2, autoscaling, clusters-keeper)
2. general new ENVs:

DASK_TLS_CA_FILE=/home/scu/.dask/dask-crt.pem
DASK_TLS_KEY=/home/scu/.dask/dask-key.pem
DASK_TLS_CERT=/home/scu/.dask/dask-crt.pem

3. director-v2 BREAKING CHANGES ENVs - ⚠️ devops
   Ensure this changed from tcp to tls protocol

COMPUTATIONAL_BACKEND_DEFAULT_CLUSTER_URL=tls://XXXXX_dask-scheduler:8786

This variable was added but contains a default because this is the only way to keep the .env simple on simcore side (it basically is made out of the dask-sidecar variables). Therefore, this would not need to be setup in ops-env

COMPUTATIONAL_BACKEND_DEFAULT_CLUSTER_AUTH='{"type":"tls","tls_ca_file":"${DASK_TLS_CERT}","tls_client_cert":"${DASK_TLS_CERT}","tls_client_key":"${DASK_TLS_KEY}"}'

4. clusters-keeper ENVs changes - ⚠️ devops (only for AWS deployments)
   These are new ENVs, they allow connecting on-demand clusters with the certificates.
   In order for the deployed EC2s to use the certificates they need to download them out of AWS ParameterStore as the certificates exceed the 16KB limit when starting an instance.

PRIMARY_EC2_INSTANCES_ATTACHED_IAM_PROFILE=AWS_SSM_READONLY_PROFILE
PRIMARY_EC2_INSTANCES_SSM_TLS_DASK_CA=/AWS/SSM/TLS_DASK_CA
PRIMARY_EC2_INSTANCES_SSM_TLS_DASK_CERT=/AWS/SSM/TLS_DASK_CERT
PRIMARY_EC2_INSTANCES_SSM_TLS_DASK_KEY=/AWS/SSM/TLS_DASK_KEY

The following is also not necessary in ops-env for same reason as explained in 2.

CLUSTERS_KEEPER_COMPUTATIONAL_BACKEND_DEFAULT_CLUSTER_AUTH='{"type":"tls","tls_ca_file":"${DASK_TLS_CERT}","tls_client_cert":"${DASK_TLS_CERT}","tls_client_key":"${DASK_TLS_KEY}"}'

5. the autoscaling now receives the following new ENVs (only for computational mode, i.e. not affecting ops env)

DASK_SCHEDULER_AUTH='{"type":"tls","tls_ca_file":"/home/scu/.dask/dask-crt.pem","tls_client_cert":"/home/scu/.dask/dask-crt.pem","tls_client_key":"/home/scu/.dask/dask-key.pem"}'

[codecov]

Codecov Report

Attention: 10 lines in your changes are missing coverage. Please review.

Comparison is base (1797692) 85.3% compared to head (c864a1f) 87.1%.

Additional details and impacted files

@@           Coverage Diff            @@
##           master   #5265     +/-   ##
========================================
+ Coverage    85.3%   87.1%   +1.7%     
========================================
  Files        1316    1316             
  Lines       53753   53813     +60     
  Branches     1170    1170             
========================================
+ Hits        45904   46912   +1008     
+ Misses       7600    6652    -948     
  Partials      249     249             

Flag	Coverage Δ
integrationtests	63.7% <88.8%> (-1.4%)	⬇️
unittests	85.1% <81.5%> (+2.0%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files	Coverage Δ
packages/aws-library/src/aws_library/ec2/client.py	100.0% <ø> (ø)
packages/aws-library/src/aws_library/ec2/models.py	88.6% <100.0%> (+0.1%)	⬆️
...ages/models-library/src/models_library/clusters.py	100.0% <100.0%> (ø)
...g/src/simcore_service_autoscaling/core/settings.py	97.9% <100.0%> (+<0.1%)	⬆️
...e_service_autoscaling/modules/auto_scaling_core.py	94.2% <ø> (ø)
...scaling/modules/auto_scaling_mode_computational.py	91.0% <100.0%> (+0.4%)	⬆️
...c/simcore_service_clusters_keeper/core/settings.py	96.0% <100.0%> (+0.2%)	⬆️
...imcore_service_clusters_keeper/modules/clusters.py	100.0% <ø> (ø)
...lusters_keeper/modules/clusters_management_core.py	100.0% <100.0%> (ø)
.../src/simcore_service_clusters_keeper/utils/dask.py	100.0% <100.0%> (ø)
... and 7 more

... and 71 files with indirect coverage changes

[pcrespov]

Cool!
As a general remark, I wonder whether all the certificates tooling (create, delete, list, ...) could be generalized for any service and not only the dask services.
If it is not such a hustle, it might become handy in the future for some other service.

[sonarqubecloud]

Quality Gate passed

Kudos, no new issues were introduced!

0 New issues
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

[mrnicegyu11]

ack for the devops changes. Looks scarry but I saw no issues conceptually