Several additional contributions to help get all dfmodules integtests working#449
Merged
bieryAtFnal merged 10 commits intoSep 12, 2025
Merged
Conversation
added 4 commits
September 11, 2025 10:39
…tweaked an expected TC fragment size in that test.
…meters in large_trigger_record_test.py
…ty_factor to simulate a nearly-full disk instead of requiring the user to create a temporary disk with limited space.
… to add ones that have been recently fixed, and to have the shorter ones run first.
Collaborator
Author
|
Added fixes for |
added 3 commits
September 11, 2025 12:10
…rs in disabled_output_test.py
…py to try to reliably get 2 data files produced.
… number of raw data files produced.
Collaborator
Author
|
Getting the It seems that the HDF5 files take up extra space (e.g. 0.6 GB) when they are being written compared to when they are closed. So, the system may find that there is not room for a second TriggerRecord in the second raw data file, but there is space for the first TR in the third raw data file. I worked around this by allowing there to be 2 or 3 raw data files produced in the test. There are still plenty of validation tests that demonstrate that the system is encountering insufficient space (e.g. required log file messages), so I believe that allowing this range is OK. |
eflumerf
approved these changes
Sep 11, 2025
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
To help get all of the
dfmodulesintegtests working, I have made a few changes. These changes are based on Eric'seflumerf/IntegrationTestUpdatesbranch.config_substitutionindisabled_output_test. It now needs to beattribute_substitution. I'm not sure how I missed that before. I also reduced the maximum expected size of TriggerCandidate fragments in kRandom triggers. For those triggers, TC fragments should always have size 128.directory_path,busy_threshold, andfree_thresholdin thelarge_trigger_record_test. These were minor oversights, and I don't expect the behavior of this test to change.insufficient_disk_space_testto use thefree_space_safety_factorto simulate the use of a nearly full disk. This will allow this test to be run more easily, without requiring users to create temporary disks of limited size.attribute_substitutioninstead ofconfig_substitutionintrmonrequestor_test.py.dfmodules_integtest_bundle.shscript to include the recently-fixed tests, and to run the shorter tests first.Testing Suggestions
With these changes, one should see 7 available tests when we run
and when we run all of the tests in the bundle script, all of them should pass (modulo the problem reported in drunc Issue #568 that affects the
maximum_file_size_testandhdf5_compression_test).Also, when we run
dbt-build --integtest dfmodules, alldfmodulesintegtests should pass (modulo thedruncissue).