Extend optional saving of test output #12184

saintstack · 2025-06-05T05:49:13Z

  Extend archiving failed logs in TestHarness2/Joshua Framework

   Unified logging output so easy to download: per run there
   is a th_run.ENSEMBLE_ID directory of logs.

   On failure if TH_ARCHIVE_LOGS_ON_FAILURE is set, logs are
   NOT cleaned up at the end of the test run: e.g.

     j start  --env TH_UNSEED_CHECK_RATIO=1.0 --env TH_ARCHIVE_LOGS_ON_FAILURE=true --tarball  /root/BulkDumpingS3.tar.gz --max-run 1

   In particular, added capture of BOTH log sets when doing the
   determinacy check so can compare the runs.

   Amended the joshua_logtool so doesn't run by default if it finds rocksdb
   log entries -- the current configuration; instead you have to ask it to run.
   It will bundle up all log files rather than just trace files (including logs
   from both runs for the determinacy test). It does NOT include the
   joshua.xml for now because joshua_logtool.py runs before this file is
   generated (hard to refactor).

   Added diagnostic logging and tracing around the joshua test runner
   context. Added capture of stderr/out so can easier debug failed test/test
   infrastructure (handy because I kept breaking tes runner). Retain the v1
   (original) test xml result reporting even after messing w/ the xml report
   to make a  proper xml document of ALL the test output.

   * contrib/Joshua/README.md
     Some detail on scripts directory and TestHarness2 integration

   * contrib/Joshua/scripts/correctnessTest.sh
     Add error handling, logging, and cleanup around test launch.

   * contrib/TestHarness2/README.md
     Documentation on operation, output structure, and archival modes
     for test harness ('th') including how to use joshua_logtool.py.

   * contrib/TestHarness2/test_harness/app.py
     Add capture of stdout/err into a well-defined location.
     Adds XML manipulation trying to make sure the XML output
     is same as it was in V1 -- the current, original output
     -- adding filtering and insertion. If fatal error,
     return an 'xml' fatal error. Made more robust
     against the unexpected (of which there was plenty

   * contrib/TestHarness2/test_harness/config.py
     Fixup. Added --joshua-output-dir option. Made
     kill seconds configurable (was 30 minutes
     which was painful when app was deadlocking).
     Added configurable log-level and whether to
     save logs on failure.

   * contrib/TestHarness2/test_harness/run.py
     Improved test execution and result handling
     Capture both determinacy test log sets.

   * contrib/TestHarness2/test_harness/summarize.py
     Hack on XML handling -- filtering -- and then
     work to make it v1/original compatible.

   * contrib/joshua_logtool.py
    Integrate joshua_logtool. Make it pick up all logs
    including double logging by determinacy test.
    Add doc.

foundationdb-ci · 2025-06-05T06:14:56Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 29997c5
Duration 0:25:30
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T06:26:27Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 29997c5
Duration 0:37:03
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T06:27:15Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 29997c5
Duration 0:37:49
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T06:35:43Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 29997c5
Duration 0:46:17
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T06:38:04Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 29997c5
Duration 0:48:39
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T06:50:06Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 29997c5
Duration 1:00:43
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T06:50:45Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 29997c5
Duration 1:01:20
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-05T20:13:53Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: f1d7eb3
Duration 0:24:29
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T20:26:34Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: f1d7eb3
Duration 0:37:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T20:34:46Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: f1d7eb3
Duration 0:45:21
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T20:36:29Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: f1d7eb3
Duration 0:47:03
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T20:38:31Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: f1d7eb3
Duration 0:49:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T20:49:22Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: f1d7eb3
Duration 0:59:59
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-05T20:52:43Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: f1d7eb3
Duration 1:03:21
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-06-06T00:45:26Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 1c8a80b
Duration 0:25:02
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-06T00:57:42Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: 1c8a80b
Duration 0:37:14
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-06T00:58:33Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 1c8a80b
Duration 0:38:07
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-06T01:07:22Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 1c8a80b
Duration 0:46:55
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-06T01:09:47Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 1c8a80b
Duration 0:49:21
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-06T01:20:51Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: 1c8a80b
Duration 1:00:26
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-06-06T01:22:15Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 1c8a80b
Duration 1:01:49
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

jzhou77 · 2025-06-11T15:41:11Z

contrib/Joshua/README.md

@@ -21,3 +21,9 @@ We use Joshua to simulate failures modes at the network, machine, and datacenter
 For a while, there was an informal competition within the engineering team to design failures that found the toughest bugs and issues the most easily. After a period of one-upsmanship, the reigning champion is called "swizzle-clogging". To swizzle-clog, you first pick a random subset of nodes in the cluster. Then, you "clog" (stop) each of their network connections one by one over a few seconds. Finally, you unclog them in a random order, again one by one, until they are all up. This pattern seems to be particularly good at finding deep issues that only happen in the rarest real-world cases.

 Joshua's success has surpassed our expectation and has been vital to our engineering team. It seems unlikely that we would have been able to build FoundationDB without this technology.
+
+*   `scripts/`: This directory contains shell scripts that serve as entry points for running tests. Joshua invokes these scripts, which then set up the environment and execute the test runner.
+    *   **`correctnessTest.sh`**: This is the primary script for running correctness tests. It is responsible for invoking the Python-based `TestHarnessV2` and passing it the necessary configuration. It also handles the creation and cleanup of temporary output directories.


Suggested change

* **`correctnessTest.sh`**: This is the primary script for running correctness tests. It is responsible for invoking the Python-based `TestHarnessV2` and passing it the necessary configuration. It also handles the creation and cleanup of temporary output directories.

* **`correctnessTest.sh`**: This is the primary script for running correctness tests. It is responsible for invoking the Python-based `TestHarness2` and passing it the necessary configuration. It also handles the creation and cleanup of temporary output directories.

foundationdb-ci · 2025-07-07T19:28:18Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: a89fbaa
Duration 0:26:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-07T19:43:37Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: a89fbaa
Duration 0:41:30
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-07T19:45:17Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: a89fbaa
Duration 0:43:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-07T19:48:27Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: a89fbaa
Duration 0:46:20
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T00:05:03Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: cf507a5
Duration 0:25:15
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T00:18:59Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: cf507a5
Duration 0:39:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T00:23:52Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: cf507a5
Duration 0:44:03
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T00:28:22Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: cf507a5
Duration 0:48:34
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T00:28:35Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: cf507a5
Duration 0:48:46
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T00:38:03Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: cf507a5
Duration 0:58:13
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-09T00:42:13Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: cf507a5
Duration 1:02:22
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T01:35:27Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: b0d393b
Duration 0:25:31
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T01:49:06Z

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Commit ID: b0d393b
Duration 0:39:10
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T01:56:25Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: b0d393b
Duration 0:46:30
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T01:58:42Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: b0d393b
Duration 0:48:47
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T01:59:20Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: b0d393b
Duration 0:49:27
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T02:10:30Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: b0d393b
Duration 1:00:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-09T02:13:31Z

Result of foundationdb-pr-macos on macOS Ventura 13.x

Commit ID: b0d393b
Duration 1:03:38
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

Unified logging output so easy to download: per run there is a th_run.ENSEMBLE_ID directory of logs. On failure if TH_ARCHIVE_LOGS_ON_FAILURE is set, logs are NOT cleaned up at the end of the test run: e.g. j start --env TH_UNSEED_CHECK_RATIO=1.0 --env TH_ARCHIVE_LOGS_ON_FAILURE=true --tarball /root/BulkDumpingS3.tar.gz --max-run 1 In particular, added capture of BOTH log sets when doing the determinacy check so can compare the runs. Amended the joshua_logtool so doesn't run by default if it finds rocksdb log entries -- the current configuration; instead you have to ask it to run. It will bundle up all log files rather than just trace files (including logs from both runs for the determinacy test). It does NOT include the joshua.xml for now because joshua_logtool.py runs before this file is generated (hard to refactor). Added diagnostic logging and tracing around the joshua test runner context. Added capture of stderr/out so can easier debug failed test/test infrastructure (handy because I kept breaking tes runner). Retain the v1 (original) test xml result reporting even after messing w/ the xml report to make a proper xml document of ALL the test output. * contrib/Joshua/README.md Some detail on scripts directory and TestHarness2 integration * contrib/Joshua/scripts/correctnessTest.sh Add error handling, logging, and cleanup around test launch. * contrib/TestHarness2/README.md Documentation on operation, output structure, and archival modes for test harness ('th') including how to use joshua_logtool.py. * contrib/TestHarness2/test_harness/app.py Add capture of stdout/err into a well-defined location. Adds XML manipulation trying to make sure the XML output is same as it was in V1 -- the current, original output -- adding filtering and insertion. If fatal error, return an 'xml' fatal error. Made more robust against the unexpected (of which there was plenty while trying to alter how this joshua test runner works). * contrib/TestHarness2/test_harness/config.py Fixup. Added --joshua-output-dir option. Made kill seconds configurable (was 30 minutes which was painful when app was deadlocking). Added configurable log-level and whether to save logs on failure. * contrib/TestHarness2/test_harness/run.py Improved test execution and result handling Capture both determinacy test log sets. * contrib/TestHarness2/test_harness/summarize.py Hack on XML handling -- filtering -- and then work to make it v1/original compatible. * contrib/joshua_logtool.py Integrate joshua_logtool. Make it pick up all logs including double logging by determinacy test. Add doc.

saintstack · 2025-07-09T18:26:03Z

This patch has gotten too complicated; it changes too much. It is also hard to prove it is working properly. I can run it at scale but there is a failure rate that seems in excess of what the current framework does. These odd failures are hard to debug too. Let me make this a draft for now. I'll break out some of the change in here -- the documentation as separate PRs because these are useful w/o disturbing operations.

foundationdb-ci · 2025-07-09T18:35:13Z

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Commit ID: 342675b
Duration 0:29:16
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T18:54:58Z

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Commit ID: 342675b
Duration 0:48:58
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T19:09:37Z

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Commit ID: 342675b
Duration 1:03:37
Result: ✅ SUCCEEDED
Error: N/A
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)
Cluster Test Logs zip file of the test logs (available for 30 days)

foundationdb-ci · 2025-07-09T19:29:26Z

Result of foundationdb-pr on Linux RHEL 9

Commit ID: 342675b
Duration 1:23:26
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

foundationdb-ci · 2025-07-09T19:36:16Z

Result of foundationdb-pr-clang on Linux RHEL 9

Commit ID: 342675b
Duration 1:30:21
Result: ❌ FAILED
Error: Error while executing command: if python3 -m joshua.joshua list --stopped | grep ${ENSEMBLE_ID} | grep -q 'pass=10[0-9][0-9][0-9]'; then echo PASS; else echo FAIL && exit 1; fi. Reason: exit status 1
Build Log terminal output (available for 30 days)
Build Workspace zip file of the working directory (available for 30 days)

saintstack · 2025-07-17T20:55:04Z

#12261 adds saving of test output in a simpler way. Resolving this PR in favor of the former.

saintstack marked this pull request as draft June 5, 2025 05:49

jzhou77 reviewed Jun 17, 2025

View reviewed changes

saintstack force-pushed the TH_ARCHIVE_LOGS_ON_FAILURE branch from 1c8a80b to a89fbaa Compare July 7, 2025 19:01

saintstack requested a review from xis19 July 7, 2025 19:03

saintstack marked this pull request as ready for review July 7, 2025 19:04

michael stack added 4 commits July 8, 2025 22:03

Ensure config is same for both runs of determinacy tests

6e7ecfd

Ensure stats are same for both runs

7686d48

Stats still not right

342675b

saintstack force-pushed the TH_ARCHIVE_LOGS_ON_FAILURE branch from b0d393b to 342675b Compare July 9, 2025 18:05

saintstack marked this pull request as draft July 9, 2025 18:26

saintstack mentioned this pull request Jul 10, 2025

Add doc, more options around log archiving, and facility debugging joshua #12231

Merged

5 tasks

saintstack closed this Jul 17, 2025

saintstack mentioned this pull request Jul 17, 2025

When the determinacy check runs, save both sets of logs. #12261

Merged

	* `correctnessTest.sh`: This is the primary script for running correctness tests. It is responsible for invoking the Python-based `TestHarnessV2` and passing it the necessary configuration. It also handles the creation and cleanup of temporary output directories.
	* `correctnessTest.sh`: This is the primary script for running correctness tests. It is responsible for invoking the Python-based `TestHarness2` and passing it the necessary configuration. It also handles the creation and cleanup of temporary output directories.

Extend optional saving of test output #12184

Extend optional saving of test output #12184

Uh oh!

Conversation

saintstack commented Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 5, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 6, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 6, 2025

Result of foundationdb-pr-macos-m1 on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 6, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 6, 2025

Result of foundationdb-pr on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jun 6, 2025

Result of foundationdb-pr-clang-arm on Linux CentOS 7

Uh oh!

foundationdb-ci commented Jun 6, 2025

Result of foundationdb-pr-macos on macOS Ventura 13.x

Uh oh!

foundationdb-ci commented Jun 6, 2025

Result of foundationdb-pr-cluster-tests on Linux RHEL 9

Uh oh!

jzhou77 Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

foundationdb-ci commented Jul 7, 2025

Result of foundationdb-pr-clang-ide on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 7, 2025

Result of foundationdb-pr-clang on Linux RHEL 9

Uh oh!

foundationdb-ci commented Jul 7, 2025

saintstack commented Jun 5, 2025 •

edited

Loading