Skip to content

Added an hdf5_compression_test and restored a couple of the existing tests#415

Merged
bieryAtFnal merged 18 commits into
patch/fddaq-v5.3.xfrom
kbiery/various_integtest_changes
Jun 9, 2025
Merged

Added an hdf5_compression_test and restored a couple of the existing tests#415
bieryAtFnal merged 18 commits into
patch/fddaq-v5.3.xfrom
kbiery/various_integtest_changes

Conversation

@bieryAtFnal
Copy link
Copy Markdown
Collaborator

The main point of this PR is to add an integration test to verify the correct operation of the HDF5 compression code. The new test is called hdf5_compression_test.py.

As part of improving the dfmodules integtest generally, the v4 multi_output_file_test.py has been split into max_file_size_test.py and multiple_data_writers_test.py, both of which are now operational in v5.x. Several of the other existing tests still need to be updated for v5.

There were also some read-ability changes to integtests, copied from ones in the daqsystemtest repo. And, there were some updates to the dfmodules_integtest_bundle.sh script, copying improvements from the similar script in daqsystemtest.

To test these changes, I recommend running the 3 available integtests multiple times, such as with a command like dfmodules_integtest_bundle.sh -N 3 --stop-on-fail.

I'll post instructions soon for creating a software area to test these changes.

Kurt Biery and others added 18 commits May 20, 2025 10:58
… v5 software and infrastructure, and to add in features that we like from the daqsystemtest bundle script.
…and large_trigger_record_test.py a bit. These integtests may not be fully operational in v5, though.
…lect its current purpose, updated some variable names inside it, and added the cleanup of TP Stream files after the test completes.
…lect its current purpose, updated some variable names inside it, and added the cleanup of TP Stream files after the test completes.
…ata writers in a single Dataflow App. (This functionality accidentally got lost in the conversion form v4 to v5.)
…mods to this integtest to get it to run more reliably (e.g. larger tp_input queue).
….py, to try to avoid occasional failures to push data onto that queue.
@bieryAtFnal bieryAtFnal requested a review from andrewmogan June 4, 2025 15:30
@bieryAtFnal
Copy link
Copy Markdown
Collaborator Author

Not surprisingly, if we locally-modify the compression_levels specified in the hdf5_compression_test so that compression is turned off, the test does not complete successfully.

This is good news, and a nice confirmation that the test is doing some meaningful checks, IMO.

@bieryAtFnal
Copy link
Copy Markdown
Collaborator Author

Here are suggested instructions for creating a software area and running a test:

DATE_PREFIX=`date '+%d%b'`
TIME_SUFFIX=`date '+%H%M'`

source /cvmfs/dunedaq.opensciencegrid.org/setup_dunedaq.sh
setup_dbt fddaq-v5.3.1
dbt-create -b candidate fddaq-v5.3.2-rc2-a9 ${DATE_PREFIX}FDv5.3.2rc2Test_${TIME_SUFFIX}
cd ${DATE_PREFIX}FDv5.3.2rc2Test_${TIME_SUFFIX}/sourcecode

git clone https://github.com/DUNE-DAQ/daqconf.git -b kbiery/nonzero_slots_for_PDS
git clone https://github.com/DUNE-DAQ/daqsystemtest.git -b patch/fddaq-v5.3.x
git clone https://github.com/DUNE-DAQ/dfmodules.git -b kbiery/various_integtest_changes
cd ..

dbt-workarea-env
dbt-build -j 12
dbt-workarea-env

daqconf_set_connectivity_service_port local-1x1-config config/daqsystemtest/example-configs.data.xml

dfmodules_integtest_bundle.sh --stop-on-fail

Copy link
Copy Markdown
Contributor

@andrewmogan andrewmogan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I ran the dfmodules test bundle as suggested and can confirm all the tests pass. I also see that, if I set the compression_level to 0 for either stream, I get the appropriate error corresponding to too many files being created due to exceeding the maximum file size. Thanks for doing this!

@bieryAtFnal bieryAtFnal merged commit 770d077 into patch/fddaq-v5.3.x Jun 9, 2025
1 check passed
@bieryAtFnal bieryAtFnal deleted the kbiery/various_integtest_changes branch June 9, 2025 16:19
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants