🚀 Pre-release master -> staging_Athena_1

[pcrespov]

In preparation for pre-release. Here an initial (incomplete) list of tasks to prepare before pre-releasing:

• Draft changelog from commits list (see docs/releasing-workflow-instructions.md)
• Check important changes 🚨
• Devops check (⚠️ devops)
• e2e testing check
• Pre-release summary
• Pre-release assessment

---

Motivation

• first round of new features from Athena sprint
• hotfix deployment in staging which currently contains the outcome of Switz sprint

e2e testing check

• Mon 21: master consistent failures in parallel test. Waiting for @GitHK fixes
• Tue 22: needs action. got worse. Alarm messages in batman channel during Mon. Follow up on agent by @GitHK .
• Wed 23: all fixes to parallel tests in place. It has been cleaned by @GitHK . 🚑️ hotfix release staging_switzer_5.
• Thu 24: ...
• Fri 25:
• Mon 28: unstable (@pcrespov and @sanderegg checks)
• Mon.5:

Commits (in order)

• ✅ Is3569/flaky public api test (#3634)
• ✨Improve phone registration (#3624)
• ✨ Rocket-lite Navigation Bar review (#3619)
• Revert "⬆️ Maintenance: Upgrade python-socketio + flakyness (#3622)" (#3630)
• ⬆️ Maintenance: Upgrade python-socketio + flakyness (#3622)
• ✨ Is686/apiserver api 0.4.2: studies ports (#3623)
• ✨ Is686/webserver api 0.12: project ports metadata (#3621)
• ✨Autoscaling: automatically cleanup nodes from the docker swarm (#3617)
• ✨ React to study:open 409 error code (#3618)
• ✨Group Templates by Tag (#3587)
• ✨ Allow to limit number of opened studies per user (🗃️ DB change) (#3615)
• ✨ Allow to override dy-service resources (#3608)
  (Mon, Nov.28)
• ✨ Migrate autoscaling (⚠️ devops) (#3566) 🚨
• ♻️ Replaces SMTP_SENDER by the products email (⚠️ devops) (#3576)
• upgrade traefik to 2.9.5 latest version (#3605)
• ♻️ Simple readable agent logs (⚠️ devops) (#3599)
• flaky pylint: moves helpers to pytest_simcore (#3603) [ 📌 v1.39.2, 1.38.5]
• Request service frontend after successful ping (#3598)
• ♻️ making agent output more readable (#3594)
• ✨ FE: Tags response includes access-rights (#3595)
• ✨ Is765/tags response include access-rights (#3591) 🚨
  (Wed, Nov.23)
• ♻️ Dy-services: Changed dierctor-v2 -> dy-sidecar API retry policy (#3583) [ 📌 staging_switzer_5]
• 🐛 Dy-Services: Stopping container without starting them no longer raises error (#3589) [ 📌 staging_switzer_5]
• 🐛 Dy-Services:No more /health errors when starting sidecars (#3586) [ 📌 staging_switzer_5]
• 🐛 optional instructions (#3593) 🚨 [📌 staging_switzer_5]
• ♻️ 2FA: Is3318/resend sms: refactoring (1/2) (#3577)
• ✨ Tags: Is765/tags for groups (⚠️ devops) (#3584)
  (Fri, Nov.18)
• 🐛 Agent: agent rclone configuration fixes (#3580) 🚨 [ 📌 staging_switzer_5 ]
• 🔨 tests e2e: Increase S4L startup time (#3575)
• 🐛 Products: Is765/fix default product (⚠️ devops ) (#3552) 🚨
• 🐛 Agent: running in production now has access to rclone (#3571) [ 📌 staging_switzer_5]
• ♻️ Use common RabbitMQ client (⚠️ devops) (#3502) 🚨
• ♻️ UI: Extends products: frontend (#3557)
• 🐛 UI: Force progress value to be between 1 and 99 (#3560) [ 📌 staging_switzer_5]
• ♻️ Rerevise docker networks dk (bis) (⚠️ devops) (#3564) [ 📌 staging_switzer_5]
• ♻️ dont add /var/lib/docker/volumes in global docker-compose file (#3563) [📌 staging_switzer_5]
• ✨ Products: Is765/extends products plugin (⚠️ devops) (#3540) 🚨

Draft Changelog

## Features (Added / Changed / Fixed)

- ✨ 2FA: Improve phone registration (#3624)
- ♻️  2FA: Resend sms (1/2) (#3577) (ITISFoundation/osparc-issues#3318)
- ✨ API: Studies ports. apiserver api `0.4.2` (#3623), webserver api `0.12` (#3621) (ITISFoundation/osparc-issues#686)
- ✨ Autoscaling: automatically cleanup nodes from the docker swarm (#3617)
- ✨ Autoscaling: Migrate autoscaling  (⚠️ devops) (#3566) 🚨
- ✨ UI:Rocket-lite Navigation Bar review (#3619)
- ✨ UI: React to study:open 409 error code (#3618)
- ✨ UI: Request service frontend after successful ping (#3598) 
- 🐛 UI/App mode: optional instructions (#3593) 🚨  [📌  ``staging_switzer_5``]
- 🐛 UI: Force progress value to be between 1 and 99 (#3560) [ 📌  ``staging_switzer_5``]
- ✨ Tags for groups w/ access-rights (#3595)(#3591) (⚠️ devops) (#3584) (#3587) 🚨
- ✨ Resources: Allow to limit number of opened studies per user (🗃️ DB change) (#3615)
- ✨ Resources: Allow to override dy-service resources (#3608)
- ✨ Products: extends products plugin (⚠️ devops)   (#3540) (ITISFoundation/osparc-issues#765) 🚨
- ♻️ Products: Replaces SMTP_SENDER by the products email (⚠️ devops) (#3576) 
- 🐛 Products: fix default product (⚠️ devops ) (#3552) (ITISFoundation/osparc-issues#765) 🚨 
- ♻️  Products/UI: Extends products (#3557)
- ♻️ Agent: Simple readable agent logs (⚠️ devops) (#3599)
- ♻️ Agent: making agent output more readable (#3594)
- 🐛 Agent: agent rclone configuration fixes (#3580) 🚨 [ 📌 ``staging_switzer_5`` ]
- 🐛 Agent: running in production now has access to rclone (#3571) [ 📌  ``staging_switzer_5``]
- ♻️  Dy-services: Changed dierctor-v2 -> dy-sidecar API retry policy (#3583) [ 📌  ``staging_switzer_5``]
- 🐛 Dy-Services: Stopping container without starting them no longer raises error (#3589) [ 📌  ``staging_switzer_5``]
- 🐛 Dy-Services:No more /health errors when starting sidecars (#3586) [ 📌  ``staging_switzer_5``]


## Platform & Infrastructure (Security / Maintenance)

- 🔨CI: Is3569/flaky public api test (#3634)
- 🔨 CI: flaky pylint: moves helpers to pytest_simcore (#3603)  [ 📌  ``v1.39.2``, ``1.38.5``]
- 🔨 tests e2e: Increase S4L startup time (#3575)
- Revert "⬆️ Maintenance: Upgrade python-socketio + flakyness (#3622)" (#3630)
- ⬆️ Maintenance: Upgrade python-socketio + flakyness (#3622)
- ⬆️ Maintenance: Upgrade traefik to 2.9.5 latest version (#3605)
- ♻️ Maintenance: Use common RabbitMQ client  (⚠️ devops) (#3502) 🚨
- ♻️ Rerevise docker networks dk (bis)  (⚠️ devops) (#3564) [ 📌 ``staging_switzer_5``]
- ♻️ dont add /var/lib/docker/volumes in global docker-compose file (#3563) [📌 ``staging_switzer_5``]


**Legend**

- ✨ New feature
- 🐛 Fixes bugs
- ♻️ Refactors code
- ⬆️ Upgrades dependencies
- 🔒️ Fixes security issues
- 🔨 Adds or updates development scripts or CI.
- 📌 Cherry-picked to production or staging
- 🚨 Has to be target/user testsed ,✅ target/User tests done
- ⚠️ Changes relevant to devops
- 🗃️ Changes in the database

Devops check and changes (⚠️ devops)

• Roll out deployment-agent v0.10.4
• Assert admin-panels service has docker-socket mounted

Pre-release summary

• what:
  • make release-staging name=Athena version=1
  • https://github.com/ITISFoundation/osparc-simcore/releases/new?prerelease=1&target=master&tag=staging_Athena1&title=Staging%20Athena1
• who: @Surfict @mrnicegyu11
• when: TUESDAY Dec.6, morning

Pre-release assessment

[pcrespov]

🚑️ hotfix release to staging (see [📌 staging_switzer5 ] above ) https://github.com/ITISFoundation/osparc-simcore/releases/tag/staging_switzer5

[mrnicegyu11]

staging_switzer5 has been rolled out.
Observed issues:

• Known but unfixed bug in the deployment agent required manual restarts of the deployment agent
• CRITICAL Portainer did not pull the latest image of the director-v2, inspection of the vcs-tag showed an old release, this would have likely gone undetected if we didnt check, a "update and pull latest image" of director-v2 via the portainer GUI resolved the issue. Cause unknown, needs monitoring upon future releases

[pcrespov]

Hotfix release 1.38.5: https://github.com/ITISFoundation/osparc-simcore/releases/tag/v1.38.5

[mrnicegyu11]

DevOps Changes assessment
For ITISFoundation/osparc-simcore#3566: Make sure the autoscaling service is set to replica=0 for now

For the rest, it is clearly stated in the PRs.
Green light from my side, but this release requires downtime and some work on the devops side. No strict blockers in there though.

[sanderegg]

DevOps Changes assessment For ITISFoundation/osparc-simcore#3566: Make sure the autoscaling service is set to replica=0 for now

For the rest, it is clearly stated in the PRs. Green light from my side, but this release requires downtime and some work on the devops side. No strict blockers in there though.

@mrnicegyu11 : the autoscaling does nothing if not configured. and this is tested. so nothing to worry about there.

[Surfict]

Rolled out today.

• Athena_1 was based on the last commit in master, who had a failed CI. I rolled out Athena_2 based on the commit 3701d49f7261aaa720aa08d9b0fd70d2eb564136
• Some difficulty due to the new way ops repo function, but I was able to do the release properly.
• A new branch in ops-environment has been created (staging_athena_1) from staging_switzer5.
• The maintenance page can't work with the new deployment agent system. We need either a major refactoring or a new way to use it (as we do with TIP for example), this need to be discussed with @mrnicegyu11
• @mrnicegyu11 Could we check tomorrow the second point of the Ops changes here

Everything else went smoothly.