Skip to content

Several additional contributions to help get all dfmodules integtests working#449

Merged
bieryAtFnal merged 10 commits into
eflumerf/IntegrationTestUpdatesfrom
kbiery/integtest_updates
Sep 12, 2025
Merged

Several additional contributions to help get all dfmodules integtests working#449
bieryAtFnal merged 10 commits into
eflumerf/IntegrationTestUpdatesfrom
kbiery/integtest_updates

Conversation

@bieryAtFnal
Copy link
Copy Markdown
Collaborator

@bieryAtFnal bieryAtFnal commented Sep 11, 2025

Description

To help get all of the dfmodules integtests working, I have made a few changes. These changes are based on Eric's eflumerf/IntegrationTestUpdates branch.

  • I fixed a use of config_substitution in disabled_output_test. It now needs to be attribute_substitution. I'm not sure how I missed that before. I also reduced the maximum expected size of TriggerCandidate fragments in kRandom triggers. For those triggers, TC fragments should always have size 128.
  • I added explicit setting of the directory_path, busy_threshold, and free_threshold in the large_trigger_record_test. These were minor oversights, and I don't expect the behavior of this test to change.
  • I re-worked the insufficient_disk_space_test to use the free_space_safety_factor to simulate the use of a nearly full disk. This will allow this test to be run more easily, without requiring users to create temporary disks of limited size.
  • I fixed the use of attribute_substitution instead of config_substitution in trmonrequestor_test.py.
  • I updated the list of tests in the dfmodules_integtest_bundle.sh script to include the recently-fixed tests, and to run the shorter tests first.

Testing Suggestions

With these changes, one should see 7 available tests when we run

dfmodules_integtest_bundle.sh --help

and when we run all of the tests in the bundle script, all of them should pass (modulo the problem reported in drunc Issue #568 that affects the maximum_file_size_test and hdf5_compression_test).

Also, when we run dbt-build --integtest dfmodules, all dfmodules integtests should pass (modulo the drunc issue).

Kurt Biery added 4 commits September 11, 2025 10:39
…tweaked an expected TC fragment size in that test.
…ty_factor to simulate a nearly-full disk instead of requiring the user to create a temporary disk with limited space.
… to add ones that have been recently fixed, and to have the shorter ones run first.
@bieryAtFnal
Copy link
Copy Markdown
Collaborator Author

Added fixes for trmonrequestor_test.py.

@bieryAtFnal
Copy link
Copy Markdown
Collaborator Author

Getting the insufficient_disk_space_test to consistently produce the same number of raw data files turned out to be a little complicated.

It seems that the HDF5 files take up extra space (e.g. 0.6 GB) when they are being written compared to when they are closed.

So, the system may find that there is not room for a second TriggerRecord in the second raw data file, but there is space for the first TR in the third raw data file.

I worked around this by allowing there to be 2 or 3 raw data files produced in the test. There are still plenty of validation tests that demonstrate that the system is encountering insufficient space (e.g. required log file messages), so I believe that allowing this range is OK.

@bieryAtFnal bieryAtFnal merged commit 5d40343 into eflumerf/IntegrationTestUpdates Sep 12, 2025
@bieryAtFnal bieryAtFnal deleted the kbiery/integtest_updates branch September 12, 2025 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants