✨ Is3565/service environs substitution [part 1] (⚠️ devops)

[pcrespov]

What do these changes do?

Custom service metadata includes some entries with identifiers instead of concrete values. These identifiers have to be resolved at runtime into concrete values depending on the context. For instance in this metadata

    compose_spec:
        service-one:
            init: true
            image: ${SIMCORE_REGISTRY}/${DOCKER_IMAGE_NAME}-dynamic-sidecar-compose-spec:${SERVICE_VERSION}

the image value has to be resolved by substituting the identifiers SIMCORE_REGISTRY and DOCKER_IMAGE_NAME with the corresponding values.

To achieve this we distinguish two part. The first part (in this PR) addresses the problem of string substitution in a unified and standard way. The second part (in following PRs) tackles how these identifiers are resolved into values depending on the runtime context (e.g. active project, user, product, service, etc).

Changes highlights

• ✨ models_library.utils.string_substitutions module
  • uses python standard Template strings
  • adds legacy conversions (previous identifiers are converted into new $-like identifiers)
• ♻️ servicelib.docker_compose.replace_env_vars_in_compose_spec
  - moved from servicelib to models_library.utils.docker_compose
  - implemented with new TemplateText
• ♻️ EXTRA: valid changes on invitations from discarded PR ✨ Is3683/traefik invitations rules (⚠️ devops) #3787
• ⚠️ devops: adds INVITATIONS_HOST variable in .env-devel

Related issue/s

• Implementing Pass deployment variables to service containers #3565: identifiers parsing, conversion and substitution

How to test

• tested against all metadata in deployed registry (AWS-production)
• Check pytest packages/models-library/tests/test_utils_string_substitution.py

Checklist

• make version-*
• Unit tests for the changes exist
• Runs in the swarm
• Documentation reflects the changes

[codecov]

Codecov Report

Merging #3803 (f0837bf) into master (e1a286d) will increase coverage by 2.3%.
The diff coverage is 60.0%.

@@            Coverage Diff            @@
##           master   #3803      +/-   ##
=========================================
+ Coverage    84.8%   87.1%    +2.3%     
=========================================
  Files         924     463     -461     
  Lines       39507   23891   -15616     
  Branches      810     136     -674     
=========================================
- Hits        33531   20828   -12703     
+ Misses       5764    3014    -2750     
+ Partials      212      49     -163     

Flag	Coverage Δ
integrationtests	61.5% <100.0%> (-5.1%)	⬇️
unittests	85.0% <60.0%> (+3.0%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../src/simcore_service_dynamic_sidecar/core/utils.py	85.5% <ø> (-0.2%)	⬇️
...mcore_service_dynamic_sidecar/core/docker_utils.py	76.8% <33.3%> (-1.5%)	⬇️
...v2/modules/dynamic_sidecar/docker_compose_specs.py	69.5% <100.0%> (-29.6%)	⬇️
...s/dynamic_sidecar/docker_service_specs/settings.py	41.5% <100.0%> (-39.5%)	⬇️
...k/src/simcore_sdk/node_ports_v2/port_validation.py	47.9% <0.0%> (-48.0%)	⬇️
...ules/dynamic_sidecar/docker_service_specs/proxy.py	53.8% <0.0%> (-46.2%)	⬇️
...s/dynamic_sidecar/scheduler/_core/_events_utils.py	54.0% <0.0%> (-40.4%)	⬇️
...modules/dynamic_sidecar/scheduler/_core/_events.py	59.5% <0.0%> (-37.7%)	⬇️
... and 487 more

[mrnicegyu11]

no review just an ack for the devops changes :) good luck and thanks a lot!

[sanderegg]

All good. maybe check my minor comments. Thanks!

[GitHK]

👍 Nice!
Please find some concerns below.

[codeclimate]

Code Climate has analyzed commit f0837bf and detected 0 issues on this pull request.

View more on Code Climate.

[sonarcloud]

Kudos, SonarCloud Quality Gate passed!   

0 Bugs
0 Vulnerabilities
0 Security Hotspots
0 Code Smells

No Coverage information
0.0% Duplication