Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor Unified configuration; remove unused references #11770

Merged
merged 3 commits into from
Oct 24, 2023

Conversation

amaltaro
Copy link
Contributor

@amaltaro amaltaro commented Oct 18, 2023

Fixes #11695

Status

ready

Description

List of changes is:

  • added a bin/adhoc script to inject only the needed set of Unified configuration (current snapshot)
  • simple aesthetic changes in MSManager
  • remove Unified configuration from MSTransferor/RequestInfo, as it has apparently been deprecated for that service
  • create a new exception for UnifiedSchema and move it under REST/Error
  • stop fetching the Unified configuration in AuxCacheUpdateTasks ReqMgr2 CherryPy thread
  • added validation on the server side for unifiedconfig endpoint. Plus moved a few other things around.
  • provided a new function validateUnifiedConfig to validate user input when creating and/or updating the Unified configuration in central CouchDB

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

None

External dependencies / deployment changes

None

@amaltaro
Copy link
Contributor Author

And with the script provided in this PR, I have just updated/cleaned the Unified configuration in testbed, as can be seen in:
https://cmsweb-testbed.cern.ch/reqmgr2/data/unifiedconfig/config

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 1 warnings and errors that must be fixed
    • 14 warnings
    • 76 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 20 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14564/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 1 warnings and errors that must be fixed
    • 14 warnings
    • 76 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 20 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14565/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 6 new failures
    • 1 tests no longer failing
    • 3 tests added
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 6 warnings and errors that must be fixed
    • 18 warnings
    • 141 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 70 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14566/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

Oups, let me fix those unit tests before any code review.

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 3 tests added
  • Python3 Pylint check: failed
    • 8 warnings and errors that must be fixed
    • 18 warnings
    • 179 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 79 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14567/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

test this please

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 1 tests no longer failing
    • 3 tests added
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 8 warnings and errors that must be fixed
    • 18 warnings
    • 179 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 79 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14568/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

unit test failure is unrelated. This PR is ready for a review.

@amaltaro
Copy link
Contributor Author

And to not delay it any further, I have just updated the Unified configuration in central production to have the latest updates provided in the Unified repository.

}

if __name__ == '__main__':
url = 'cmsweb-testbed.cern.ch'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please do not hard-code options, it is better if you'll use sys.argv to pass around it as an input option. If it is not provided then you can fall back to specific default (like this one)>


print(f"Updating unified configuration in {url} with content: {pformat(DOC)}")
conn = http.client.HTTPSConnection(url,
cert_file=os.getenv('X509_USER_PROXY'),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

before making a call, please add block to check this env parameters, and if they are empty please provide meaningful message. The script should exit with non zero status in this case.

print("Response status: %s\tResponse reason: %s" % (resp.status, resp.reason))
if hasattr(resp.msg, "x-error-detail"):
print("Error message: %s" % resp.msg["x-error-detail"])
sys.exit(1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you exit here your connection will remain open, please refactor the code with with clause to automatically close the connection

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 3 tests added
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 8 warnings and errors that must be fixed
    • 18 warnings
    • 178 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 79 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14572/artifact/artifacts/PullRequestReport.html

Remove Unified update from AuxCacheUpdateTasks thread

Add schema and validation to the unifiedconfig endpoint

Standardize error messages

new line

separating AuxValidation_t unit tests

Use safe.kwargs in the REST server
Apply Valentins suggestions to injectUnified.py
fix conflict in unit test

adjust unit tests

AuxValidation_t unit tests
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 1 tests no longer failing
    • 3 tests added
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 8 warnings and errors that must be fixed
    • 18 warnings
    • 178 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 79 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14574/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor Author

I actually planned to remove that script under "bin/adhoc-scripts/injectUnified.py", but it turns out we might actually need it for our testX kubernetes clusters, so I decided to keep it around and I have also applied the suggestions that you made, @vkuznet .

Initial description has been updated and it's now ready for another review. The unit test failure is unrelated - and actually unstable.

@amaltaro amaltaro requested a review from vkuznet October 23, 2023 21:13
@amaltaro amaltaro merged commit 1c861c3 into dmwm:master Oct 24, 2023
1 of 4 checks passed
@amaltaro
Copy link
Contributor Author

I just noticed that the Unified configuration in testbed and production had all the previous parameters (in addition to the supported ones). After cross-checking the default configuration in WMCore against the P&R gitlab's one, I pushed the up-to-date data structure into testbed and production, from vocms0193 (latest agent):

python3 apps/wmagentpy3/bin/adhoc-scripts/injectUnified.py -c cmsweb.cern.ch

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Change Unified configuration url for AuxCacheUpdateTasks CherryPy thread
3 participants