Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add a generic script for deploying WMAgent inside a virtual environment. #11624

Merged

Conversation

todor-ivanov
Copy link
Contributor

@todor-ivanov todor-ivanov commented Jun 27, 2023

Fixes #11617

Status

ready

Description

With the current script we provide a method for implementing the same set of steps as done during the deployment process inside the WMAgent docker image, but inside a well contained virtual environment. The end result should be a runtime environment equal to the one inside the Docker image.
NOTE: All Os dependencies need to be resolved in advance

NOTE: Here follow some instructions on how to deploy and run the agent:

./wmagent-venv-deploy.sh  -d /data/WMAgent.venv3 -t 2.2.1 -y
source /data/WMAgent.venv3/bin/activate
source $WMA_ENV_FILE
/data/WMAgent.venv3/run.sh
manage status

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

dmwm/CMSKubernetes#1364

External dependencies / deployment changes

dmwm/CMSKubernetes#1364

Once the above PR in CMSkubernetes is merged we should proceed and fix the following lines in this script before merging it:
TODO:
To change

deployRepo=https://github.com/todor-ivanov/CMSKubernetes.git
deployBranch=Fix_WMAgentPyPi_DcokerImage 

to

deployRepo=https://github.com/dmwm/CMSKubernetes.git 
deployBranch=master

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14282/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14289/artifact/artifacts/PullRequestReport.html

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@todor-ivanov as we discussed this week, I'd rather have someone from T0 giving this script a try and providing us with feedback.
Nonetheless, I left a few relevant comments along the code that you might want to address before collecting feedback.

deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
@todor-ivanov
Copy link
Contributor Author

todor-ivanov commented Jun 30, 2023

Hi @amaltaro I think I have addressed all your comments. Could you please take another look
As of the T0 side, @germanfgv is already testing the PR. He was reporting some OS dependencies like:

ERROR: No matching distribution found for Jinja2~=3.1.2

I will give him the list of all the OS packages that may be needed for this setup.

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14295/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14298/artifact/artifacts/PullRequestReport.html

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@todor-ivanov please see a few extra comments inline. Once the relevant changes are made, we could then ask German to have another look and try this out.

deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
deploy/wmagent-venv-deploy.sh Outdated Show resolved Hide resolved
@todor-ivanov
Copy link
Contributor Author

todor-ivanov commented Jul 14, 2023

hi @amaltaro I have addressed all your comments and tested the script. Please take another look
@germanfgv please update your clone before your next test.

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14334/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14545/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 1 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14546/artifact/artifacts/PullRequestReport.html

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Todor, I left a comment in the code, but I didn't review most of the code.
It's still not clear to me whether we will be using this script for the agent, but if you are using it or prefer to get it merged, let's get it done.

deploy/deploy-wmagent-venv.sh Outdated Show resolved Hide resolved
@todor-ivanov
Copy link
Contributor Author

hi @LinaresToine there was a dedicated effort from both of us lately in the form of hands on meetings and redeployment && readjustments in replay agents etc. There are still things to be done of course, like the oracle functionalities, which are about to come later during the completion of the meta issue about the full containerization of the WMAgents: #11314 && #11720 , but they won't require any further deployments or tests related to the functionalities provided with this script here.

I'd like to close the issue #11617, which is basically for confirming that the so provided script here make things to happen and we are capable of having a generic WMAgent deployment with it. Which is something we did prove with you during our meetings. If you agree with me, we should ask from @amaltaro for final approval and we should merge the script and close the issue.

@amaltaro
Copy link
Contributor

amaltaro commented Dec 4, 2023

@todor-ivanov I would suggest to resolve the review I left weeks ago. Second, let's squash those commits to the most meaningful situation (i.e., it does not need to be 1 commit only, but let's squash unneeded commit messages).

Change mode to 755 from wmagent-venv-deploy.sh

Fix python executable link

Remove setupDependencies for run from source mode && Fix outdated Rucio links && Fix help bool flags

Substitute export command with _addWMCoreVenvVar function && Add X509 variables to the environment.

Removing serviceList && Removing setupRucio && Cloning CMSKubernetes from dmwm

Catchup with WMAgent init changes

Rename wmagent-venv-deploy to deploy-wmagent-venv.sh

ReviewComments && Fix missing tag param && Remove _realPath && Remove sudo commands
@todor-ivanov todor-ivanov force-pushed the feature_Deploy_WMAgentVenv_fix-11617 branch from 36a089a to 502b0b2 Compare December 5, 2023 09:05
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14690/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14689/artifact/artifacts/PullRequestReport.html

@todor-ivanov
Copy link
Contributor Author

@amaltaro I did address all your comments and squashed all the commits, If you find it ok now, lets merge this.

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Todor.

@amaltaro amaltaro merged commit 6b9368e into dmwm:master Dec 5, 2023
3 of 4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Create a script to deploy WMAgen in a virtual env. replicating the Docker image runtime env.
3 participants