Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Core] Unit tests fixes/improvements #40988

Merged
merged 1 commit into from Mar 8, 2023

Conversation

smuzaffar
Copy link
Contributor

Updated unit tests so that they can be run from their own separate directory. Mostly

  • convert <bin ...> to <test....>
  • use full path e.g CMSSW_BASE/src/... instead of ./src/
  • added flag USE_UNITTEST_DIR="1" to run each tests in separate dir. Once all the tests can run it their own tmp path then we can enable USE_UNITTEST_DIR="1" at project level and cleanup FWCore/Framework/test/BuildFile.xml

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 8, 2023

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-40988/34498

  • This PR adds an extra 24KB to repository

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 8, 2023

A new Pull Request was created by @smuzaffar (Malik Shahzad Muzaffar) for master.

It involves the following packages:

  • FWCore/Framework (core)

@cmsbuild, @smuzaffar, @Dr15Jones, @makortel can you please review it and eventually sign? Thanks.
@makortel, @missirol, @wddgit this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

@smuzaffar
Copy link
Contributor Author

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 8, 2023

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-6e2427/31137/summary.html
COMMIT: 8a42460
CMSSW: CMSSW_13_1_X_2023-03-07-2300/el8_amd64_gcc11
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/40988/31137/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially removed 2 lines from the logs
  • Reco comparison results: 0 differences found in the comparisons
  • DQMHistoTests: Total files compared: 49
  • DQMHistoTests: Total histograms compared: 3530381
  • DQMHistoTests: Total failures: 1
  • DQMHistoTests: Total nulls: 0
  • DQMHistoTests: Total successes: 3530358
  • DQMHistoTests: Total skipped: 22
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
  • Checked 213 log files, 164 edm output root files, 49 DQM output files
  • TriggerResults: no differences found

@makortel
Copy link
Contributor

makortel commented Mar 8, 2023

+1

Thanks @smuzaffar!

@cmsbuild
Copy link
Contributor

cmsbuild commented Mar 8, 2023

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

@perrotta
Copy link
Contributor

perrotta commented Mar 8, 2023

+1

@cmsbuild cmsbuild merged commit e7caf24 into cms-sw:master Mar 8, 2023
11 checks passed
@@ -3,10 +3,11 @@
# Pass in name and status
function die { echo $1: status $2 ; exit $2; }

LOCAL_TEST_DIR="${CMSSW_BASE}/src/FWCore/Framework/test"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Given that scram knows it is executing a test, couldn't we pass a special environment variable to the tests to let them know about where they come from? As seen here, that is a very useful thing for the test. It also avoids having to know exactly which CMSSW variable are available both in a local work area and in the IBs. It would also help if a test is moved from one directory to a different directory for maintenance reasons as one would not have to remember to update the internals of the test as well.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Dr15Jones , scram does set the following env

export SCRAM_TEST_NAME="<name-of-test>"
export SCRAM_TEST_PATH="$(CMSSW_BASE)/src/<Package/Name>/test"
export SCRAM_TEST_PACKAGE="<Package/Name>"
export SCRAM_TEST_PRE_TESTS="<test-dependencies-on-other-tests>"

so one can make use of these but the only problem is that these are set when you run tests via scram i.e. scram build runtests or scram b unittests. So test will fail if you try to run them directly

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In the previously written tests, they would also fail if not run via scram as the arguments needed for the test were embedded in the BuildFile.xml files. Therefore I do not see it as a problem to use those environment variables in the tests themselves.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not really, previously the TestHelper was setting various env e.g LOCAL_TMP_DIR, LOCAL_TEST_DIR so when you run the unit tests executable e.g. TestFWCoreFrameworkDeleteEarly then it works. Of course if you run the shell script directly then yes it would have failed.

But I have no issues if tests can make use of these SCRAM_TEST_*

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TestHelper generates LOCAL_TEST_DIR based on the 3rd argument it is passed and that argument is embedded in the BuildFile.xml file. So executing just TestFWCoreFrameworkDeleteEarly should fail with an 'invalid number of arguments' error.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ah yes, you are right

@wddgit
Copy link
Contributor

wddgit commented Mar 8, 2023

It is a trade-off. One way is easier to maintain if the test is moved to a different directory and also it is a little easier to write the shell script (less characters to type). The other way makes it easier to run the test without scram. I don't have a preference.

There is one thing that would be nice though. We could add a little to the scram documentation. If we pick Chris's choice, the documentation could give a recipe to run the test without scram (which environment variables need to be set to what and an easy recipe to copy/paste/edit that will do that). Also we should mention "scram b runtests_TestName" in the scram documentation. The biggest reason to run a test standalone is to avoid running all the tests in the directory. For FWCore/Integration, there are many. Although it is faster to run standalone even if scram only runs one test.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants