Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Changes in src/Utils and test/Utils_t to remove py2 compatibilities (issue #11421) #11618

Merged
merged 1 commit into from
Jul 20, 2023

Conversation

anpicci
Copy link
Contributor

@anpicci anpicci commented Jun 27, 2023

Fixes #11421

Status

ready

Description

Changes in Utils and Utils_t modules to remove all Py2-compatibility dependencies

Is it backward compatible (if not, which system it affects?)

YES (for python3)

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 2 warnings and errors that must be fixed
    • 18 comments to review
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded
    • 3 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14281/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 33 new failures
    • 8 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 186 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 83 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14297/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor

amaltaro commented Jul 6, 2023

test this please

@@ -299,3 +295,5 @@ def encodeUnicodeToBytesConditional(value, errors="ignore", condition=True):
if condition:
return encodeUnicodeToBytes(value, errors)
return value

makeList("")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess this line shouldn't be here(?)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, you are correct, I added it to debug and then forgot to delete :/

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anpicci Andrea, thank you for creating this pull request. I left a comment for you along the code. In addition to that:
a) can you please remove files that are likely results from unit tests, they are: ReportEmuTestFile.txt, basicWorkload, lumiTest.json, newFile.json, nose-marker.txt, nosetests.xml
b) rename this pull request, given that it now changes all the src/python/Utils package
?

Feel free to reach out to me via mattermost if you want to discuss any of these and/or need assistance along the way (at some point we will have to squash/rebase this PR as well ;)).

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 34 new failures
    • 9 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 186 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 83 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14306/artifact/artifacts/PullRequestReport.html

@anpicci anpicci changed the title MemoryCache changes to fullfil #11421 Changes in src/Utils and test/Utils_t to remove py2 compatibilities (#11421) Jul 7, 2023
@anpicci anpicci changed the title Changes in src/Utils and test/Utils_t to remove py2 compatibilities (#11421) Changes in src/Utils and test/Utils_t to remove py2 compatibilities (issue #11421) Jul 7, 2023
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 33 new failures
    • 1 tests no longer failing
    • 9 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 186 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 82 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14307/artifact/artifacts/PullRequestReport.html

@amaltaro
Copy link
Contributor

test this please

@anpicci anpicci requested a review from amaltaro July 13, 2023 19:16
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 186 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 82 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14330/artifact/artifacts/PullRequestReport.html

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anpicci thanks for fixing those unit tests.
In addition to removing the basicWorkload file from this PR, I left some minor comments along the code for your consideration.

The last part will be to rebase/squash all of your commits into only 2:

  1. with the src/* changes
  2. with the test/* changes.
    This is one of the procedures we recommend for rebasing/squashing: https://steveklabnik.com/writing/how-to-squash-commits-in-a-github-pull-request , but if you prefer to do this together, I am happy to. Thanks again!

src/python/Utils/MemoryCache.py Outdated Show resolved Hide resolved
src/python/Utils/ProcessStats.py Outdated Show resolved Hide resolved
src/python/Utils/Signals.py Outdated Show resolved Hide resolved
src/python/Utils/TwPrint.py Outdated Show resolved Hide resolved
test/python/Utils_t/Utilities_t.py Outdated Show resolved Hide resolved
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 186 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 80 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14338/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 186 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 80 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14339/artifact/artifacts/PullRequestReport.html

Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anpicci there should be only a couple of extra changes required in this PR and we are ready to move forward with this (after a rebase/squashing). Please see my comments along the code. Thanks

src/python/Utils/MemoryCache.py Outdated Show resolved Hide resolved
basicWorkload Outdated Show resolved Hide resolved
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 187 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 79 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14342/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 187 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 79 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14343/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 11 warnings and errors that must be fixed
    • 21 warnings
    • 186 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 79 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14344/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 changes in unstable tests
  • Python3 Pylint check: failed
    • 10 warnings and errors that must be fixed
    • 21 warnings
    • 187 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 78 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14345/artifact/artifacts/PullRequestReport.html

Copy link
Contributor Author

@anpicci anpicci left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have addressed these

first changes to MemoryCache.py (and its _t counterpart) for dmwm#11421 issue

removing comments in MemoryCache.py (and its _t counterpart)

References to python2 compatibility removed from all the modules in Utils and Utils_t

Implementing comments from Alan

Fixing bug in Utils/IteratorTools.py

Fixing small flaws in Utils/IteratorTools.py

Fixing small flaws in Utils/MemoryCache.py

Deleting basicWorkload

Fixing small flaws in Utils/MemoryCache.py

Addressing Jenkins warnings
@cmsdmwmbot
Copy link

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
  • Python3 Pylint check: failed
    • 10 warnings and errors that must be fixed
    • 21 warnings
    • 187 comments to review
  • Pylint py3k check: failed
    • 10 errors and warnings that should be fixed
  • Pycodestyle check: succeeded
    • 78 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/14348/artifact/artifacts/PullRequestReport.html

@anpicci anpicci requested a review from amaltaro July 20, 2023 14:12
Copy link
Contributor

@amaltaro amaltaro left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anpicci these changes look good to me. As we discussed today, potentially some of those dictionary iterations didn't need to be casted to a list (only when the dictionary actually changes its size that we need to do so), but it should be fine as they are as well.

I also took this opportunity to update this wiki:
https://github.com/dmwm/WMCore/wiki/python3-transition#identifying-python2-code-to-be-dropped

with the information I had collected from previous PRs and the feedback you provided as well. Feel free to update it in case I've gotten anything wrong. Thanks

@amaltaro amaltaro merged commit e920d72 into dmwm:master Jul 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Convert Utils package to python3-only (remove future dependency)
3 participants