DAOS-17639 test: Detect all server fabric_ifaces #16913

phender · 2025-09-25T23:13:33Z

Launch.py will detect all of the fastest interfaces common to all the specified server hosts and use them to populate the engine fabric_iface entries if no overrides are provided in the test yaml.

Skip-unit-tests: true
Skip-fault-injection-test: true
Test-tag: IorSmall

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

Launch.py will detect all of the fastest interfaces common to all the specified server hosts and use them to populate the engine fabric_iface entries if no overrides are provided in the test yaml. Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

github-actions · 2025-09-25T23:13:51Z

Ticket title is 'Support newly named ib devices for functional tests'
Status is 'In Review'
Labels: 'testp1'
https://daosio.atlassian.net/browse/DAOS-17639

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-09-25T23:56:33Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/4/execution/node/557/log

daosbuild3 · 2025-09-26T00:06:55Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/3/execution/node/805/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-09-26T04:55:58Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/6/execution/node/747/log

daosbuild3 · 2025-09-26T06:40:18Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/5/execution/node/805/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-09-26T15:19:03Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/7/execution/node/805/log

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-09-27T07:59:45Z

Test stage Functional on EL 8.8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/10/execution/node/665/log

daosbuild3 · 2025-09-27T08:45:19Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/11/execution/node/557/log

into hendersp/DAOS-17639

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-10-03T03:38:34Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/14/execution/node/895/log

daosbuild3 · 2025-10-03T14:09:14Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/14/execution/node/954/log

daosbuild3 · 2025-10-03T20:53:46Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/14/execution/node/909/log

phender · 2025-10-06T13:12:36Z

Failures seen in https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16913/14/testReport/ are known issues or should not be related to PR changes - in all cases the servers started successfully:

2-./container/boundary.py:BoundaryTest.test_container_boundary - https://daosio.atlassian.net/browse/DAOS-18040
1-./erasurecode/multiple_rank_failure.py:EcodOnlineMultiRankFail.test_ec_multiple_rank_failure - https://daosio.atlassian.net/browse/DAOS-16339
1-./soak/smoke.py:SoakSmoke.test_soak_smoke - https://daosio.atlassian.net/browse/DAOS-18043
6-./nvme/enospace.py:NvmeEnospace.test_enospace_no_aggregation - This test suppose to fail because of DER_NOSPACEbut it got Passed
4-./recovery/pool_list_consolidation.py:PoolListConsolidationTest.test_lost_majority_ps_replicas -
19-./daos_test/suite.py:DaosCoreTest.test_daos_rebuild_simple_interactive - timeout waiting for rebuild
24-./daos_test/suite.py:DaosCoreTest.test_daos_rebuild_ec - https://daosio.atlassian.net/browse/DAOS-17657
1-./recovery/cat_recov_core.py:CatRecovCoreTest.test_daos_cat_recov_core - https://daosio.atlassian.net/browse/DAOS-17977

JohnMalmberg

I have some concerns about how robust this solution is, as on the Ice Lake systems, we usually run two HDR adapter speeds.

And we run two OPA adapters speeds currently on at least one of our Omni-Path clusters.

JohnMalmberg · 2025-10-06T14:07:04Z

src/tests/ftest/control/daos_system_query.yaml

      targets: 4
      nr_xs_helpers: 0
-      fabric_iface_port: 31416
+      fabric_iface: eth0


Is this a default template value?

There is nothing guaranteeing that and eth0 device will be active right now if a system has multiple ethernet adapters, which is the case for many of our systems.

This is documented in the Red Hat EL-6 manuals, and as of the el-8 release several systems in the lab have been observed to alternate at random which adapter is eth0 at each boot, which is not seen on the older distro versions.

You have to look up the interface properties such as its online (carrier detected) and IP subnet in order to find interfaces on a common network.

The eth and ib interface names have been deprecated from Linux support and are only available by specifying a kernel option to enable them. In the future we want to stop setting that option so that we are prepared for when the kernels stop supporting it.

This is what currently works with our VM tests/clusters. This particular test defines a dual engine config where we only have one active interface on the VM host. The simplest way to handle this outlier of a test to manually set the same expected CI interface for both engines in this and the src/tests/ftest/control/dmg_system_start.py test.

I understand this may not always be the case for our VMs, but for now it will work. The alternative approach - which was considered - involves re-working the entire test yaml replacement code to support keyword replacement. This would be a larger undertaking at this time.

I've adjusted the logic used to determine the default fabric_iface value for each engine to reuse the last known interface from the DAOS_TEST_FABRIC_IFACE env var when the index exceeds the list. This means if DAOS_TEST_FABRIC_IFACE=foo0 (or whatever is discovered on the VM server hosts) and the test requests dual engines, then engine 0 and engine 1 will both use fabric_iface: foo0. If the DAOS_TEST_FABRIC_IFACE is not set, then it will fall back to using eth0 and eth1, respectively.

JohnMalmberg · 2025-10-06T14:22:15Z

src/tests/ftest/util/network_utils.py

    try:
-        # Select the fastest active interface available by sorting the speed
-        interface = available_interfaces[sorted(available_interfaces)[-1]]
+        fastest_speed = sorted(interfaces_at_speed)[-1]


On the Mellanox HDR network, the test controller node will usually have a reported speed of 100G for the ConnectX-[5/6] interface and on Ice Lake systems the rest of the cluster will have a reported speed of 200G for the ConnectX-6 interface.

On the Omni-Path network, the test controller may have a reported speed of 56G and the rest of the cluster will have a reported speed of 100G.

On VMs, the reported network speed may be pure fiction and may not even have a value to look up or even parse as numeric value.

In this PR we are only checking the speeds of the adapters on the server hosts - excluding the launch node.

For VMs we have (currently and historically) only matched one device, so even when the speed is fictional (e.g. -1) we still get the expected device:

2025/10/02 07:28:23 DEBUG _default_interface: Detecting network devices on brd-103vm[02-09] - DAOS_TEST_FABRIC_IFACE not set 2025/10/02 07:28:23 DEBUG run_remote: Running on brd-103vm[02-09] with a 120 second timeout: grep -l 'up' /sys/class/net/*/operstate | grep -Ev '/(lo|bond)/' | sort 2025/10/02 07:28:24 DEBUG log_result_data: brd-103vm[02-09] (rc=0): /sys/class/net/eth0/operstate 2025/10/02 07:28:24 INFO get_common_interfaces: Active network interfaces detected: 2025/10/02 07:28:24 INFO get_common_interfaces: - eth0 on brd-103vm[02-09] (Common=True) 2025/10/02 07:28:24 DEBUG run_remote: Running on brd-103vm[02-09] with a 120 second timeout: cat /sys/class/net/eth0/speed 2025/10/02 07:28:24 DEBUG log_result_data: brd-103vm[02-09] (rc=0): -1 2025/10/02 07:28:24 INFO get_fastest_interfaces: Active network interface speeds on brd-103vm[02-09]: 2025/10/02 07:28:24 INFO get_fastest_interfaces: - speed: -1 => ['eth0'] 2025/10/02 07:28:24 INFO get_fastest_interfaces: Fastest interfaces detected on brd-103vm[02-09]: ['eth0'] 2025/10/02 07:28:24 DEBUG _default_interface: Found interface(s): eth0

daosbuild3 · 2025-10-06T15:18:08Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/15/execution/node/553/log

src/tests/ftest/daos_perf/large.yaml

daltonbohning · 2025-10-06T18:01:11Z

src/tests/ftest/util/network_utils.py

+    for speed in sorted(interfaces_at_speed):
+        logger.info("  - speed: %6s => %s", speed, sorted(interfaces_at_speed[speed]))


Maybe we can just sort the dictionary once upfront so we don't have to keep calling sorted?

Suggested change

for speed in sorted(interfaces_at_speed):

logger.info(" - speed: %6s => %s", speed, sorted(interfaces_at_speed[speed]))

interfaces_at_speed = {k: interfaces_at_speed[k] for k in sorted(interfaces_at_speed)}

for speed, interfaces in enumerate(interfaces_at_speed):

logger.info(" - speed: %6s => %s", speed, interfaces)

daltonbohning · 2025-10-06T18:03:10Z

src/tests/ftest/util/network_utils.py

+        fastest_speed = sorted(interfaces_at_speed)[-1]
+        fastest_interfaces = interfaces_at_speed[fastest_speed]


And then this also becomes

Suggested change

fastest_speed = sorted(interfaces_at_speed)[-1]

fastest_interfaces = interfaces_at_speed[fastest_speed]

fastest_speed, fastest_interfaces = list(interfaces_at_speed.items())[-1]

daltonbohning · 2025-10-06T18:05:46Z

src/tests/ftest/util/server_utils_params.py

+        try:
+            _defaults = os.environ.get("DAOS_TEST_FABRIC_IFACE").split(",")
+            default_interface = list(filter(None, _defaults))[index]
+        except (AttributeError, IndexError):
+            default_interface = f"eth{index}"
+        default_port = int(os.environ.get("D_PORT", 31317 + (100 * index)))


So if someone sets for example DAOS_TEST_FABRIC_IFACE=ib0 and the test is dual engine, we'll end up trying to use ib0 and eth0 by default? Maybe if you set DAOS_TEST_FABRIC_IFACE it should be expected that you pass an interface for each

If the user attempts to run a dual engine test and only sets DAOS_TEST_FABRIC_IFACE=ib0 we will run the test with a server config using:

engines: - fabric_iface: ib0 fabric_iface_port: 31317 - fabric_iface: eth1 fabric_iface_port: 31417

So, yes, they should have set DAOS_TEST_FABRIC_IFACE=ib0,ib1 in this case.

… hendersp/DAOS-17639 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-11-22T09:50:57Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/25/execution/node/925/log

daosbuild3 · 2025-11-23T04:10:14Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/25/execution/node/880/log

daosbuild3 · 2025-11-25T13:34:12Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/25/execution/node/1066/log

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-12-13T02:53:48Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/26/execution/node/917/log

daosbuild3 · 2025-12-13T13:04:46Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/26/execution/node/948/log

daosbuild3 · 2025-12-13T17:00:42Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/26/execution/node/927/log

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

daosbuild3 · 2025-12-17T12:22:00Z

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-16913/27/execution/node/948/log

JohnMalmberg

I have not got this to work with the new udev rules that I am currently testing.

Fixing that should be a minor change that can be done after this PR, so I do not want to have that be a blocker.

daosbuild3 · 2025-12-17T21:38:39Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-16913/27/testReport/

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: CatRecovCoreTest IorInterceptMultiClient Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender · 2025-12-19T16:32:20Z

The failures in https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16913/27/ are known issues:

2-./ior/small.py:IorSmall.test_ior_small -> https://daosio.atlassian.net/browse/DAOS-18189
6-./nvme/enospace.py:NvmeEnospace.test_enospace_no_aggregation -> https://daosio.atlassian.net/browse/DAOS-18232
7-./osa/offline_extend.py:OSAOfflineExtend.test_osa_offline_extend_exclude_during_rebuild -> https://daosio.atlassian.net/browse/DAOS-14570
1-./server/daos_server_restart.py:DaosServerTest.test_daos_server_reformat -> https://daosio.atlassian.net/browse/DAOS-17616
4-./erasurecode/multiple_rank_failure.py:EcodOnlineMultiRankFail.test_ec_multiple_rank_failure -> https://daosio.atlassian.net/browse/DAOS-16766
4-./ior/intercept_multi_client.py:IorInterceptMultiClient.test_ior_intercept_libpil4dfs -> https://daosio.atlassian.net/browse/DAOS-16343 (just merged the fix)
1-./soak/smoke.py:SoakSmoke.test_soak_smoke -> https://daosio.atlassian.net/browse/DAOS-18043

phender · 2025-12-22T04:02:05Z

https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-16913/28/ passed running the two test yaml files updated due to merge conflicts after #16913 (comment) - which included the fix for the 4-./ior/intercept_multi_client.py:IorInterceptMultiClient.test_ior_intercept_libpil4dfs failure.

Per #16913 Test-tag: test_rebuild_interactive Test-repeat: 3 Skip-func-hw-test-large: false Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Dalton Bohning <dalton.bohning@hpe.com>

Launch.py will detect all of the fastest interfaces common to all the specified server hosts and use them to populate the engine fabric_iface entries if no overrides are provided in the test yaml. Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender requested review from a team as code owners September 25, 2025 23:13

phender marked this pull request as draft September 25, 2025 23:14

Fix typo.

e362dda

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Resolve default provider interface input.

3aa7e60

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Adding debug logging.

95b63c2

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender added 2 commits September 26, 2025 15:30

Exclude local host from network device selection.

4894015

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: IorSmall Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Update remaining test ymal files.

5e3ed12

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender added 3 commits October 1, 2025 19:25

Merge branch 'master' into hendersp/DAOS-17639

60576f6

Merge branch 'hendersp/DAOS-17639' of https://github.com/daos-stack/daos

c8c689f

into hendersp/DAOS-17639

Force reuse of eth0 for dual-engine VM tests

f5f6880

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender marked this pull request as ready for review October 6, 2025 13:12

phender requested review from JohnMalmberg and daltonbohning October 6, 2025 13:13

JohnMalmberg reviewed Oct 6, 2025

View reviewed changes

daltonbohning reviewed Oct 6, 2025

View reviewed changes

Merge branch 'hendersp/DAOS-17639' of github.com:daos-stack/daos into…

afcc9d8

… hendersp/DAOS-17639 Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender dismissed stale reviews from JohnMalmberg and daltonbohning via afcc9d8 November 21, 2025 20:24

Merge branch 'master' into hendersp/DAOS-17639

2ea7d05

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

Merge branch 'master' into hendersp/DAOS-17639

9983126

Skip-unit-tests: true Skip-fault-injection-test: true Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender requested review from JohnMalmberg and daltonbohning December 17, 2025 04:24

daltonbohning previously approved these changes Dec 17, 2025

View reviewed changes

JohnMalmberg previously approved these changes Dec 17, 2025

View reviewed changes

Merge branch 'master' into hendersp/DAOS-17639

988491b

Skip-unit-tests: true Skip-fault-injection-test: true Test-tag: CatRecovCoreTest IorInterceptMultiClient Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>

phender dismissed stale reviews from JohnMalmberg and daltonbohning via 988491b December 19, 2025 16:22

daltonbohning approved these changes Dec 19, 2025

View reviewed changes

phender added the forced-landing The PR has known failures or has intentionally reduced testing, but should still be landed. label Dec 22, 2025

daltonbohning requested a review from a team December 22, 2025 18:49

daltonbohning merged commit 4acf5a3 into master Dec 22, 2025
33 checks passed

daltonbohning deleted the hendersp/DAOS-17639 branch December 22, 2025 18:49

daltonbohning mentioned this pull request Dec 22, 2025

DAOS-18025 test: add ftest/rebuild/interactive.py #17171

Merged

6 tasks

		for speed in sorted(interfaces_at_speed):
		logger.info(" - speed: %6s => %s", speed, sorted(interfaces_at_speed[speed]))

-    for speed in sorted(interfaces_at_speed):
-        logger.info("  - speed: %6s => %s", speed, sorted(interfaces_at_speed[speed]))
+    interfaces_at_speed = {k: interfaces_at_speed[k] for k in sorted(interfaces_at_speed)}
+    for speed, interfaces in enumerate(interfaces_at_speed):
+        logger.info("  - speed: %6s => %s", speed, interfaces)

		fastest_speed = sorted(interfaces_at_speed)[-1]
		fastest_interfaces = interfaces_at_speed[fastest_speed]

	fastest_speed = sorted(interfaces_at_speed)[-1]
	fastest_interfaces = interfaces_at_speed[fastest_speed]
	fastest_speed, fastest_interfaces = list(interfaces_at_speed.items())[-1]

DAOS-17639 test: Detect all server fabric_ifaces #16913

DAOS-17639 test: Detect all server fabric_ifaces #16913

Uh oh!

Conversation

phender commented Sep 25, 2025

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions bot commented Sep 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Sep 25, 2025

Uh oh!

daosbuild3 commented Sep 26, 2025

Uh oh!

daosbuild3 commented Sep 26, 2025

Uh oh!

daosbuild3 commented Sep 26, 2025

Uh oh!

daosbuild3 commented Sep 26, 2025

Uh oh!

daosbuild3 commented Sep 27, 2025

Uh oh!

daosbuild3 commented Sep 27, 2025

Uh oh!

daosbuild3 commented Oct 3, 2025

Uh oh!

daosbuild3 commented Oct 3, 2025

Uh oh!

daosbuild3 commented Oct 3, 2025

Uh oh!

phender commented Oct 6, 2025

Uh oh!

JohnMalmberg left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phender Oct 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

phender Oct 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Oct 6, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Nov 22, 2025

Uh oh!

daosbuild3 commented Nov 23, 2025

Uh oh!

daosbuild3 commented Nov 25, 2025

Uh oh!

daosbuild3 commented Dec 13, 2025

Uh oh!

daosbuild3 commented Dec 13, 2025

Uh oh!

daosbuild3 commented Dec 13, 2025

Uh oh!

daosbuild3 commented Dec 17, 2025

Uh oh!

JohnMalmberg left a comment

github-actions bot commented Sep 25, 2025 •

edited

Loading

phender Oct 7, 2025 •

edited

Loading

phender Oct 6, 2025 •

edited

Loading

phender commented Dec 22, 2025 •

edited

Loading