flaky tests: record-replay-tests #1347
Description
2025-08-22T18:51:48.6642011Z Current runner version: '2.328.0'
2025-08-22T18:51:48.6665355Z ##[group]Runner Image Provisioner
2025-08-22T18:51:48.6666168Z Hosted Compute Agent
2025-08-22T18:51:48.6666816Z Version: 20250821.380
2025-08-22T18:51:48.6667548Z Commit: 438d12cb4fe092871db9867344e3bc0bc5319035
2025-08-22T18:51:48.6668234Z Build Date: 2025-08-21T20:49:43Z
2025-08-22T18:51:48.6668836Z ##[endgroup]
2025-08-22T18:51:48.6669413Z ##[group]Operating System
2025-08-22T18:51:48.6669931Z Ubuntu
2025-08-22T18:51:48.6670433Z 24.04.2
2025-08-22T18:51:48.6670872Z LTS
2025-08-22T18:51:48.6671326Z ##[endgroup]
2025-08-22T18:51:48.6671762Z ##[group]Runner Image
2025-08-22T18:51:48.6672434Z Image: ubuntu-24.04
2025-08-22T18:51:48.6672946Z Version: 20250818.1.0
2025-08-22T18:51:48.6673884Z Included Software: https://github.com/actions/runner-images/blob/ubuntu24/20250818.1/images/ubuntu/Ubuntu2404-Readme.md
2025-08-22T18:51:48.6675464Z Image Release: https://github.com/actions/runner-images/releases/tag/ubuntu24%2F20250818.1
2025-08-22T18:51:48.6676432Z ##[endgroup]
2025-08-22T18:51:48.6679025Z ##[group]GITHUB_TOKEN Permissions
2025-08-22T18:51:48.6680890Z Actions: write
2025-08-22T18:51:48.6681501Z Attestations: write
2025-08-22T18:51:48.6682013Z Checks: write
2025-08-22T18:51:48.6682472Z Contents: write
2025-08-22T18:51:48.6683039Z Deployments: write
2025-08-22T18:51:48.6683554Z Discussions: write
2025-08-22T18:51:48.6684009Z Issues: write
2025-08-22T18:51:48.6684533Z Metadata: read
2025-08-22T18:51:48.6684975Z Models: read
2025-08-22T18:51:48.6685443Z Packages: write
2025-08-22T18:51:48.6685960Z Pages: write
2025-08-22T18:51:48.6686460Z PullRequests: write
2025-08-22T18:51:48.6687749Z RepositoryProjects: write
2025-08-22T18:51:48.6688379Z SecurityEvents: write
2025-08-22T18:51:48.6689016Z Statuses: write
2025-08-22T18:51:48.6689470Z ##[endgroup]
2025-08-22T18:51:48.6691559Z Secret source: Actions
2025-08-22T18:51:48.6692231Z Prepare workflow directory
2025-08-22T18:51:48.7104507Z Prepare all required actions
2025-08-22T18:51:48.7142457Z Getting action download info
2025-08-22T18:51:49.1161234Z Download action repository 'actions/download-artifact@v5' (SHA:634f93cb2916e3fdff6788551b99b062d0335ce0)
2025-08-22T18:51:50.0288836Z Complete job name: record-replay-tests (record_replay)
2025-08-22T18:51:50.0762685Z ##[group]Checking docker version
2025-08-22T18:51:50.0775154Z ##[command]/usr/bin/docker version --format '{{.Server.APIVersion}}'
2025-08-22T18:51:50.1298152Z '1.48'
2025-08-22T18:51:50.1310732Z Docker daemon API version: '1.48'
2025-08-22T18:51:50.1311527Z ##[command]/usr/bin/docker version --format '{{.Client.APIVersion}}'
2025-08-22T18:51:50.1461585Z '1.48'
2025-08-22T18:51:50.1475727Z Docker client API version: '1.48'
2025-08-22T18:51:50.1481708Z ##[endgroup]
2025-08-22T18:51:50.1485106Z ##[group]Clean up resources from previous jobs
2025-08-22T18:51:50.1490419Z ##[command]/usr/bin/docker ps --all --quiet --no-trunc --filter "label=8ee656"
2025-08-22T18:51:50.1627503Z ##[command]/usr/bin/docker network prune --force --filter "label=8ee656"
2025-08-22T18:51:50.1755627Z ##[endgroup]
2025-08-22T18:51:50.1756168Z ##[group]Create local container network
2025-08-22T18:51:50.1766181Z ##[command]/usr/bin/docker network create --label 8ee656 github_network_3e27553cc8cd4602b03c417f2cd7173a
2025-08-22T18:51:50.2238815Z 037aa4a2f05661fb3d8ee0e2307e1ae9be3d40ba0eb48fcb8bff93881528a959
2025-08-22T18:51:50.2257461Z ##[endgroup]
2025-08-22T18:51:50.2280922Z ##[group]Starting job container
2025-08-22T18:51:50.2300725Z ##[command]/usr/bin/docker pull gcr.io/o1labs-192920/mina-daemon:3.2.0-beta2-939b08d-noble-devnet
2025-08-22T18:51:50.9777834Z 3.2.0-beta2-939b08d-noble-devnet: Pulling from o1labs-192920/mina-daemon
2025-08-22T18:51:50.9779823Z 32f112e3802c: Pulling fs layer
2025-08-22T18:51:50.9780715Z dfe67868a319: Pulling fs layer
2025-08-22T18:51:50.9781497Z 30cf3dfe7632: Pulling fs layer
2025-08-22T18:51:50.9782288Z e8a7de2e6dc6: Pulling fs layer
2025-08-22T18:51:50.9783086Z 00fa16e46870: Pulling fs layer
2025-08-22T18:51:50.9783879Z f09ab99b425f: Pulling fs layer
2025-08-22T18:51:50.9785111Z 83e0f91e24d9: Pulling fs layer
2025-08-22T18:51:50.9785942Z 4f4fb700ef54: Pulling fs layer
2025-08-22T18:51:50.9786768Z a0fd226144a8: Pulling fs layer
2025-08-22T18:51:50.9787879Z b79554546ccc: Pulling fs layer
2025-08-22T18:51:50.9788707Z 4a522a956ec9: Pulling fs layer
2025-08-22T18:51:50.9789538Z 2c35b6e76971: Pulling fs layer
2025-08-22T18:51:50.9790369Z 1fc0a6651c3b: Pulling fs layer
2025-08-22T18:51:50.9791198Z 658ed508e8f9: Pulling fs layer
2025-08-22T18:51:50.9792106Z 00fa16e46870: Waiting
2025-08-22T18:51:50.9792883Z f09ab99b425f: Waiting
2025-08-22T18:51:50.9793720Z 83e0f91e24d9: Waiting
2025-08-22T18:51:50.9794464Z e8a7de2e6dc6: Waiting
2025-08-22T18:51:50.9795447Z 4f4fb700ef54: Waiting
2025-08-22T18:51:50.9796305Z 2c35b6e76971: Waiting
2025-08-22T18:51:50.9797548Z a0fd226144a8: Waiting
2025-08-22T18:51:50.9798413Z b79554546ccc: Waiting
2025-08-22T18:51:50.9799296Z 1fc0a6651c3b: Waiting
2025-08-22T18:51:50.9800253Z 658ed508e8f9: Waiting
2025-08-22T18:51:50.9801187Z 4a522a956ec9: Waiting
2025-08-22T18:51:51.4436758Z 30cf3dfe7632: Verifying Checksum
2025-08-22T18:51:51.4440173Z 30cf3dfe7632: Download complete
2025-08-22T18:51:51.4619349Z dfe67868a319: Verifying Checksum
2025-08-22T18:51:51.4621152Z dfe67868a319: Download complete
2025-08-22T18:51:51.6412541Z 32f112e3802c: Verifying Checksum
2025-08-22T18:51:51.6414223Z 32f112e3802c: Download complete
2025-08-22T18:51:52.1893256Z e8a7de2e6dc6: Verifying Checksum
2025-08-22T18:51:52.1893948Z e8a7de2e6dc6: Download complete
2025-08-22T18:51:52.5781383Z 83e0f91e24d9: Verifying Checksum
2025-08-22T18:51:52.5781979Z 83e0f91e24d9: Download complete
2025-08-22T18:51:52.8452790Z 32f112e3802c: Pull complete
2025-08-22T18:51:52.9277806Z 4f4fb700ef54: Verifying Checksum
2025-08-22T18:51:52.9278449Z 4f4fb700ef54: Download complete
2025-08-22T18:51:53.3353566Z a0fd226144a8: Download complete
2025-08-22T18:51:53.5805693Z 00fa16e46870: Verifying Checksum
2025-08-22T18:51:53.5806526Z 00fa16e46870: Download complete
2025-08-22T18:51:53.6491971Z f09ab99b425f: Verifying Checksum
2025-08-22T18:51:53.6492514Z f09ab99b425f: Download complete
2025-08-22T18:51:53.6659137Z b79554546ccc: Verifying Checksum
2025-08-22T18:51:53.6659613Z b79554546ccc: Download complete
2025-08-22T18:51:53.8764272Z 4a522a956ec9: Verifying Checksum
2025-08-22T18:51:53.8764945Z 4a522a956ec9: Download complete
2025-08-22T18:51:53.9667210Z 2c35b6e76971: Verifying Checksum
2025-08-22T18:51:53.9667877Z 2c35b6e76971: Download complete
2025-08-22T18:51:53.9788217Z 1fc0a6651c3b: Verifying Checksum
2025-08-22T18:51:54.1735104Z 1fc0a6651c3b: Download complete
2025-08-22T18:51:54.1735726Z 658ed508e8f9: Verifying Checksum
2025-08-22T18:51:54.1736200Z 658ed508e8f9: Download complete
2025-08-22T18:51:54.2951628Z dfe67868a319: Pull complete
2025-08-22T18:51:54.4984633Z 30cf3dfe7632: Pull complete
2025-08-22T18:51:55.9032943Z e8a7de2e6dc6: Pull complete
2025-08-22T18:52:12.9498829Z 00fa16e46870: Pull complete
2025-08-22T18:52:18.0914354Z f09ab99b425f: Pull complete
2025-08-22T18:52:18.1091611Z 83e0f91e24d9: Pull complete
2025-08-22T18:52:18.1180803Z 4f4fb700ef54: Pull complete
2025-08-22T18:52:18.1361926Z a0fd226144a8: Pull complete
2025-08-22T18:52:18.1444768Z b79554546ccc: Pull complete
2025-08-22T18:52:18.1532439Z 4a522a956ec9: Pull complete
2025-08-22T18:52:18.1649825Z 2c35b6e76971: Pull complete
2025-08-22T18:52:18.1782460Z 1fc0a6651c3b: Pull complete
2025-08-22T18:52:18.1874256Z 658ed508e8f9: Pull complete
2025-08-22T18:52:18.1908896Z Digest: sha256:c6ea0658de6edac47c8cb773a13c686b7426f8f7aca8001bed80041737d1e838
2025-08-22T18:52:18.1923279Z Status: Downloaded newer image for gcr.io/o1labs-192920/mina-daemon:3.2.0-beta2-939b08d-noble-devnet
2025-08-22T18:52:18.1932932Z gcr.io/o1labs-192920/mina-daemon:3.2.0-beta2-939b08d-noble-devnet
2025-08-22T18:52:18.2012378Z ##[command]/usr/bin/docker create --name 67bc7a8cc36f4bc0a6651266fff18134_gcrioo1labs192920minadaemon320beta2939b08dnobledevnet_b6f848 --label 8ee656 --workdir /__w/mina-rust/mina-rust --network github_network_3e27553cc8cd4602b03c417f2cd7173a -e "HOME=/github/home" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work":"/__w" -v "/home/runner/actions-runner/cached/externals":"/__e":ro -v "/home/runner/work/_temp":"/__w/_temp" -v "/home/runner/work/_actions":"/__w/_actions" -v "/opt/hostedtoolcache":"/__t" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" --entrypoint "tail" gcr.io/o1labs-192920/mina-daemon:3.2.0-beta2-939b08d-noble-devnet "-f" "/dev/null"
2025-08-22T18:52:18.2251437Z f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3
2025-08-22T18:52:18.2276101Z ##[command]/usr/bin/docker start f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3
2025-08-22T18:52:18.3790258Z f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3
2025-08-22T18:52:18.3810582Z ##[command]/usr/bin/docker ps --all --filter id=f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3 --filter status=running --no-trunc --format "{{.ID}} {{.Status}}"
2025-08-22T18:52:18.3922940Z f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3 Up Less than a second
2025-08-22T18:52:18.3940862Z ##[command]/usr/bin/docker inspect --format "{{range .Config.Env}}{{println .}}{{end}}" f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3
2025-08-22T18:52:18.4053635Z HOME=/github/home
2025-08-22T18:52:18.4054251Z GITHUB_ACTIONS=true
2025-08-22T18:52:18.4054577Z CI=true
2025-08-22T18:52:18.4055050Z PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2025-08-22T18:52:18.4055590Z SUFFIX=
2025-08-22T18:52:18.4055861Z MINA_DEB=mina-devnet
2025-08-22T18:52:18.4056194Z DEBIAN_FRONTEND=noninteractive
2025-08-22T18:52:18.4056581Z USE_GKE_GCLOUD_AUTH_PLUGIN=True
2025-08-22T18:52:18.4057135Z MINA_TIME_OFFSET=0
2025-08-22T18:52:18.4074255Z ##[endgroup]
2025-08-22T18:52:18.4083057Z ##[group]Waiting for all services to be ready
2025-08-22T18:52:18.4084760Z ##[endgroup]
2025-08-22T18:52:18.4266372Z ##[group]Run apt-get update && \
2025-08-22T18:52:18.4266696Z �[36;1mapt-get update && \�[0m
2025-08-22T18:52:18.4267261Z �[36;1m apt-get install -y --no-install-recommends libssl3t64 && \�[0m
2025-08-22T18:52:18.4267624Z �[36;1m apt-get clean && \�[0m
2025-08-22T18:52:18.4267864Z �[36;1m rm -rf /var/lib/apt/lists/*�[0m
2025-08-22T18:52:18.4271360Z shell: sh -e {0}
2025-08-22T18:52:18.4271819Z env:
2025-08-22T18:52:18.4271994Z CARGO_TERM_COLOR: always
2025-08-22T18:52:18.4272212Z RUST_BACKTRACE: full
2025-08-22T18:52:18.4272401Z MINA_PANIC_ON_BUG: true
2025-08-22T18:52:18.4272596Z CARGO_INCREMENTAL: 1
2025-08-22T18:52:18.4272864Z RUSTFLAGS: -C overflow-checks=off -C debug-assertions=off
2025-08-22T18:52:18.4273170Z MINA_DISCOVERY_FILTER_ADDR: false
2025-08-22T18:52:18.4273397Z ##[endgroup]
2025-08-22T18:52:18.5256723Z Ign:1 http://localhost:8080 noble InRelease
2025-08-22T18:52:18.6845941Z Get:2 http://archive.ubuntu.com/ubuntu noble InRelease [256 kB]
2025-08-22T18:52:18.7710307Z Get:3 http://security.ubuntu.com/ubuntu noble-security InRelease [126 kB]
2025-08-22T18:52:18.8873971Z Get:4 https://packages.cloud.google.com/apt cloud-sdk InRelease [1620 B]
2025-08-22T18:52:19.1147195Z Get:5 http://archive.ubuntu.com/ubuntu noble-updates InRelease [126 kB]
2025-08-22T18:52:19.2231176Z Get:6 http://security.ubuntu.com/ubuntu noble-security/restricted amd64 Packages [2066 kB]
2025-08-22T18:52:19.2345966Z Get:7 http://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
2025-08-22T18:52:19.2665585Z Get:8 https://packages.cloud.google.com/apt cloud-sdk/main all Packages [1797 kB]
2025-08-22T18:52:19.3421690Z Get:9 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages [19.3 MB]
2025-08-22T18:52:19.5252395Z Ign:1 http://localhost:8080 noble InRelease
2025-08-22T18:52:19.6238240Z Get:10 https://packages.cloud.google.com/apt cloud-sdk/main amd64 Packages [4064 kB]
2025-08-22T18:52:19.6563165Z Get:11 http://security.ubuntu.com/ubuntu noble-security/main amd64 Packages [1363 kB]
2025-08-22T18:52:19.6918216Z Get:12 http://security.ubuntu.com/ubuntu noble-security/universe amd64 Packages [1137 kB]
2025-08-22T18:52:19.7436426Z Get:13 http://security.ubuntu.com/ubuntu noble-security/multiverse amd64 Packages [23.0 kB]
2025-08-22T18:52:19.9614610Z Get:14 http://archive.ubuntu.com/ubuntu noble/main amd64 Packages [1808 kB]
2025-08-22T18:52:19.9899124Z Get:15 http://archive.ubuntu.com/ubuntu noble/restricted amd64 Packages [117 kB]
2025-08-22T18:52:19.9910263Z Get:16 http://archive.ubuntu.com/ubuntu noble/multiverse amd64 Packages [331 kB]
2025-08-22T18:52:19.9929348Z Get:17 http://archive.ubuntu.com/ubuntu noble-updates/multiverse amd64 Packages [45.2 kB]
2025-08-22T18:52:19.9933383Z Get:18 http://archive.ubuntu.com/ubuntu noble-updates/universe amd64 Packages [1458 kB]
2025-08-22T18:52:20.0259216Z Get:19 http://archive.ubuntu.com/ubuntu noble-updates/main amd64 Packages [1713 kB]
2025-08-22T18:52:20.0637853Z Get:20 http://archive.ubuntu.com/ubuntu noble-updates/restricted amd64 Packages [2193 kB]
2025-08-22T18:52:20.0961374Z Get:21 http://archive.ubuntu.com/ubuntu noble-backports/main amd64 Packages [48.8 kB]
2025-08-22T18:52:20.0966183Z Get:22 http://archive.ubuntu.com/ubuntu noble-backports/universe amd64 Packages [35.6 kB]
2025-08-22T18:52:21.5258334Z Ign:1 http://localhost:8080 noble InRelease
2025-08-22T18:52:25.5266487Z Err:1 http://localhost:8080 noble InRelease
2025-08-22T18:52:25.5267396Z Could not connect to localhost:8080 (127.0.0.1). - connect (111: Connection refused)
2025-08-22T18:52:25.5433784Z Fetched 38.2 MB in 7s (5435 kB/s)
2025-08-22T18:52:26.2718024Z Reading package lists...
2025-08-22T18:52:26.2824199Z W: Failed to fetch http://localhost:8080/dists/noble/InRelease Could not connect to localhost:8080 (127.0.0.1). - connect (111: Connection refused)
2025-08-22T18:52:26.2825860Z W: Some index files failed to download. They have been ignored, or old ones used instead.
2025-08-22T18:52:27.0240687Z Reading package lists...
2025-08-22T18:52:27.2006087Z Building dependency tree...
2025-08-22T18:52:27.2006544Z Reading state information...
2025-08-22T18:52:27.3585311Z libssl3t64 is already the newest version (3.0.13-0ubuntu3.5).
2025-08-22T18:52:27.3585989Z 0 upgraded, 0 newly installed, 0 to remove and 9 not upgraded.
2025-08-22T18:52:27.3927976Z ##[group]Run actions/download-artifact@v5
2025-08-22T18:52:27.3928273Z with:
2025-08-22T18:52:27.3928514Z pattern: tests*-ffbf4dd5068dc343c57fef5bb8deeea7ecb1ed52
2025-08-22T18:52:27.3928820Z merge-multiple: true
2025-08-22T18:52:27.3929024Z repository: o1-labs/mina-rust
2025-08-22T18:52:27.3929239Z run-id: 17163091085
2025-08-22T18:52:27.3929421Z env:
2025-08-22T18:52:27.3929585Z CARGO_TERM_COLOR: always
2025-08-22T18:52:27.3929782Z RUST_BACKTRACE: full
2025-08-22T18:52:27.3929963Z MINA_PANIC_ON_BUG: true
2025-08-22T18:52:27.3930154Z CARGO_INCREMENTAL: 1
2025-08-22T18:52:27.3930402Z RUSTFLAGS: -C overflow-checks=off -C debug-assertions=off
2025-08-22T18:52:27.3930718Z MINA_DISCOVERY_FILTER_ADDR: false
2025-08-22T18:52:27.3930948Z ##[endgroup]
2025-08-22T18:52:27.3968178Z ##[command]/usr/bin/docker exec f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3 sh -c "cat /etc/*release | grep ^ID"
2025-08-22T18:52:28.0079459Z Found 4 artifact(s)
2025-08-22T18:52:28.0080503Z Filtering artifacts by pattern 'tests*-ffbf4dd5068dc343c57fef5bb8deeea7ecb1ed52'
2025-08-22T18:52:28.0101413Z Preparing to download the following artifacts:
2025-08-22T18:52:28.0102485Z - tests-webrtc-ffbf4dd5068dc343c57fef5bb8deeea7ecb1ed52 (ID: 3831149189, Size: 283930319, Expected Digest: sha256:91d6dda447596e2e78d6cf2aeb8a983fa9121bb0c4554d28d2758afae48212c2)
2025-08-22T18:52:28.0103653Z - tests-ffbf4dd5068dc343c57fef5bb8deeea7ecb1ed52 (ID: 3831101113, Size: 399659233, Expected Digest: sha256:41c1771f27c91a5719625584214d2054535f4b33a62492b0ae7076bd67c61e99)
2025-08-22T18:52:28.2217465Z Redirecting to blob download url: https://productionresultssa11.blob.core.windows.net/actions-results/f594777a-2ded-435f-9b9a-6b79c3b51c92/workflow-job-run-018cf060-7d93-5cec-9bc2-160759e3c9a6/artifacts/38b70ed734fdb0f2060087230338eefd9d0509a41b2a52e0227c2992bcff02a4.zip
2025-08-22T18:52:28.2219849Z Starting download of artifact to: /__w/mina-rust/mina-rust
2025-08-22T18:52:28.3215277Z Redirecting to blob download url: https://productionresultssa11.blob.core.windows.net/actions-results/f594777a-2ded-435f-9b9a-6b79c3b51c92/workflow-job-run-a46bc1d3-7227-531e-aef3-2f6906785d1d/artifacts/bb31fb5c300d18e38dff3c8e8d351efa0d2846d07dfdcae7947510cdf4670acf.zip
2025-08-22T18:52:28.3217211Z Starting download of artifact to: /__w/mina-rust/mina-rust
2025-08-22T18:52:28.5907884Z (node:210) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
2025-08-22T18:52:28.5909489Z (Use `node --trace-deprecation ...` to show where the warning was created)
2025-08-22T18:52:43.4158342Z SHA256 digest of downloaded artifact is 91d6dda447596e2e78d6cf2aeb8a983fa9121bb0c4554d28d2758afae48212c2
2025-08-22T18:52:43.4159052Z Artifact download completed successfully.
2025-08-22T18:52:44.8583168Z SHA256 digest of downloaded artifact is 41c1771f27c91a5719625584214d2054535f4b33a62492b0ae7076bd67c61e99
2025-08-22T18:52:44.8583816Z Artifact download completed successfully.
2025-08-22T18:52:44.8584157Z Total of 2 artifact(s) downloaded
2025-08-22T18:52:44.8590471Z Download artifact has finished successfully
2025-08-22T18:52:44.8744477Z ##[group]Run chmod +x ./record_replay
2025-08-22T18:52:44.8744794Z �[36;1mchmod +x ./record_replay�[0m
2025-08-22T18:52:44.8745145Z shell: sh -e {0}
2025-08-22T18:52:44.8745333Z env:
2025-08-22T18:52:44.8745507Z CARGO_TERM_COLOR: always
2025-08-22T18:52:44.8745723Z RUST_BACKTRACE: full
2025-08-22T18:52:44.8745928Z MINA_PANIC_ON_BUG: true
2025-08-22T18:52:44.8746137Z CARGO_INCREMENTAL: 1
2025-08-22T18:52:44.8746415Z RUSTFLAGS: -C overflow-checks=off -C debug-assertions=off
2025-08-22T18:52:44.8746749Z MINA_DISCOVERY_FILTER_ADDR: false
2025-08-22T18:52:44.8747284Z ##[endgroup]
2025-08-22T18:52:44.9347837Z ##[group]Run ./record_replay --test-threads=1
2025-08-22T18:52:44.9348218Z �[36;1m./record_replay --test-threads=1�[0m
2025-08-22T18:52:44.9348569Z shell: sh -e {0}
2025-08-22T18:52:44.9348747Z env:
2025-08-22T18:52:44.9348915Z CARGO_TERM_COLOR: always
2025-08-22T18:52:44.9349118Z RUST_BACKTRACE: full
2025-08-22T18:52:44.9349315Z MINA_PANIC_ON_BUG: true
2025-08-22T18:52:44.9349518Z CARGO_INCREMENTAL: 1
2025-08-22T18:52:44.9349781Z RUSTFLAGS: -C overflow-checks=off -C debug-assertions=off
2025-08-22T18:52:44.9350097Z MINA_DISCOVERY_FILTER_ADDR: false
2025-08-22T18:52:44.9350328Z ##[endgroup]
2025-08-22T18:52:44.9889617Z
2025-08-22T18:52:44.9889827Z running 2 tests
2025-08-22T18:53:09.5850150Z test record_replay_block_production ... FAILED
2025-08-22T18:57:36.3638732Z test record_replay_bootstrap ... ok
2025-08-22T18:57:36.3639195Z
2025-08-22T18:57:36.3639341Z failures:
2025-08-22T18:57:36.3639564Z
2025-08-22T18:57:36.3639908Z ---- record_replay_block_production stdout ----
2025-08-22T18:57:36.3646260Z 2025-08-22T18:52:46.388334Z WARN mina_core::network: time: "1755888766388325143", kind: "network", "no network config initialized, using default config", config: "devnet"
2025-08-22T18:57:36.3647895Z 2025-08-22T18:52:46.391011Z INFO mina_tree::proofs::circuit_blobs: time: "1755888766391009691", kind: "ledger proofs", "circuit-blobs not found locally, so fetching it...", filename: "3.0.1devnet/block_verifier_index.postcard"
2025-08-22T18:57:36.3649635Z 2025-08-22T18:52:48.878653Z INFO mina_tree::proofs::circuit_blobs: time: "1755888768878646452", kind: "ledger proofs", "caching circuit-blobs", path: "/github/home/.openmina/circuit-blobs/3.0.1devnet/block_verifier_index.postcard"
2025-08-22T18:57:36.3650965Z 2025-08-22T18:52:48.879031Z WARN mina_tree::proofs::verifiers: Cannot load verifier index: source digest verification failed, time: "1755888768879030384"
2025-08-22T18:57:36.3651790Z 2025-08-22T18:52:55.117450Z INFO mina_tree::proofs::verifiers: Stored verifier index to cache file, time: "1755888775117446563"
2025-08-22T18:57:36.3653155Z 2025-08-22T18:52:55.119919Z INFO mina_tree::proofs::circuit_blobs: time: "1755888775119918801", kind: "ledger proofs", "circuit-blobs not found locally, so fetching it...", filename: "3.0.1devnet/transaction_verifier_index.postcard"
2025-08-22T18:57:36.3654469Z 2025-08-22T18:52:57.926418Z INFO mina_tree::proofs::circuit_blobs: time: "1755888777926413826", kind: "ledger proofs", "caching circuit-blobs", path: "/github/home/.openmina/circuit-blobs/3.0.1devnet/transaction_verifier_index.postcard"
2025-08-22T18:57:36.3655570Z 2025-08-22T18:52:57.928620Z WARN mina_tree::proofs::verifiers: Cannot load verifier index: source digest verification failed, time: "1755888777928619423"
2025-08-22T18:57:36.3656372Z 2025-08-22T18:53:04.211294Z INFO mina_tree::proofs::verifiers: Stored verifier index to cache file, time: "1755888784211290110"
2025-08-22T18:57:36.3656854Z setting up rust seed nodes: 1
2025-08-22T18:57:36.3657517Z 2025-08-22T18:53:04.212792Z INFO Server::run{addr: 0.0.0.0:11000}: warp::server: listening on http://0.0.0.0:11000
2025-08-22T18:57:36.3657969Z waiting for all rust nodes to sync up
2025-08-22T18:57:36.3658223Z Non staker total balance: 23234416
2025-08-22T18:57:36.3659111Z 2025-08-22T18:53:05.890093Z INFO mina_core::log::action: time: "1755888784211311342", node_id: 2awmjihpsd9TkdaqEkvrJ2UguAsoigRtgoiKei4AxpA5EVzr6JZ, kind: "TransitionFrontierGenesisInject", summary: "Inject genesis block into the transition frontier"
2025-08-22T18:57:36.3660450Z 2025-08-22T18:53:05.890122Z INFO mina_core::log::action: time: "1755888784211311343", node_id: 2awmjihpsd9TkdaqEkvrJ2UguAsoigRtgoiKei4AxpA5EVzr6JZ, kind: "BlockProducerBestTipUpdate"
2025-08-22T18:57:36.3661553Z 2025-08-22T18:53:05.891584Z INFO mina_core::log::action: time: "1755888784211311344", node_id: 2awmjihpsd9TkdaqEkvrJ2UguAsoigRtgoiKei4AxpA5EVzr6JZ, kind: "TransactionPoolBestTipChanged"
2025-08-22T18:57:36.3662907Z 2025-08-22T18:53:05.891593Z INFO node::transaction_pool::transaction_pool_effects: time: "1755888785891593359", kind: "TransactionPoolEffectfulFetchAccounts", summary: "fetching accounts for tx pool"
2025-08-22T18:57:36.3664161Z 2025-08-22T18:53:05.893139Z INFO mina_core::log::action: time: "1755888784211311346", node_id: 2awmjihpsd9TkdaqEkvrJ2UguAsoigRtgoiKei4AxpA5EVzr6JZ, kind: "TransactionPoolBestTipChangedWithAccounts"
2025-08-22T18:57:36.3665521Z 2025-08-22T18:53:05.893169Z INFO p2p::p2p_state: time: "1755888785893168811", kind: "P2pState new", summary: "Current node's id: 12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp", peer_id_str: "12D3KooWEyoppNCUx8Yx66oV9fJnriXwCcXwDDUA2kj6vnc6iDEp"
2025-08-22T18:57:36.3667409Z 2025-08-22T18:53:05.893201Z INFO mina_core::log::action: time: "1755888784211311347", node_id: 2awmjihpsd9TkdaqEkvrJ2UguAsoigRtgoiKei4AxpA5EVzr6JZ, kind: "P2pInitializeInitialize", summary: "Initializes p2p layer", chain_id: 89ac2b7463e5f562d55222d76bb7a11c7091ac987b87951698348b8bd809324c
2025-08-22T18:57:36.3668423Z all rust nodes synced up
2025-08-22T18:57:36.3668652Z setting up normal nodes: 1
2025-08-22T18:57:36.3669063Z 2025-08-22T18:53:05.894463Z INFO Server::run{addr: 0.0.0.0:11002}: warp::server: listening on http://0.0.0.0:11002
2025-08-22T18:57:36.3669500Z waiting for all rust nodes to sync up
2025-08-22T18:57:36.3669748Z Non staker total balance: 23234416
2025-08-22T18:57:36.3670635Z 2025-08-22T18:53:06.548288Z INFO mina_core::log::action: time: "1755888784211311342", node_id: 2bF8pb8dwWicAEpuptFvSbLRrx739bdb5uGfRiKwTuRkCBSs6nR, kind: "TransitionFrontierGenesisInject", summary: "Inject genesis block into the transition frontier"
2025-08-22T18:57:36.3671938Z 2025-08-22T18:53:06.548314Z INFO mina_core::log::action: time: "1755888784211311343", node_id: 2bF8pb8dwWicAEpuptFvSbLRrx739bdb5uGfRiKwTuRkCBSs6nR, kind: "BlockProducerBestTipUpdate"
2025-08-22T18:57:36.3673042Z 2025-08-22T18:53:06.549777Z INFO mina_core::log::action: time: "1755888784211311344", node_id: 2bF8pb8dwWicAEpuptFvSbLRrx739bdb5uGfRiKwTuRkCBSs6nR, kind: "TransactionPoolBestTipChanged"
2025-08-22T18:57:36.3674324Z 2025-08-22T18:53:06.549785Z INFO node::transaction_pool::transaction_pool_effects: time: "1755888786549785638", kind: "TransactionPoolEffectfulFetchAccounts", summary: "fetching accounts for tx pool"
2025-08-22T18:57:36.3675508Z 2025-08-22T18:53:06.551273Z INFO mina_core::log::action: time: "1755888784211311346", node_id: 2bF8pb8dwWicAEpuptFvSbLRrx739bdb5uGfRiKwTuRkCBSs6nR, kind: "TransactionPoolBestTipChangedWithAccounts"
2025-08-22T18:57:36.3676861Z 2025-08-22T18:53:06.551293Z INFO p2p::p2p_state: time: "1755888786551293393", kind: "P2pState new", summary: "Current node's id: 12D3KooWHdiAxVd8uMQR1hGWXccidmfCwLqcMpGwR6QcTP6QRMuD", peer_id_str: "12D3KooWHdiAxVd8uMQR1hGWXccidmfCwLqcMpGwR6QcTP6QRMuD"
2025-08-22T18:57:36.3678803Z 2025-08-22T18:53:06.551317Z INFO mina_core::log::action: time: "1755888784211311347", node_id: 2bF8pb8dwWicAEpuptFvSbLRrx739bdb5uGfRiKwTuRkCBSs6nR, kind: "P2pInitializeInitialize", summary: "Initializes p2p layer", chain_id: 89ac2b7463e5f562d55222d76bb7a11c7091ac987b87951698348b8bd809324c
2025-08-22T18:57:36.3679816Z all rust nodes synced up
2025-08-22T18:57:36.3680040Z setting up rust snark worker nodes: 1
2025-08-22T18:57:36.3680495Z snark worker(B62qnLjgW4LAnrxkcdLc7Snb49qx6aP5qsmPsp6ueZN4XPMC621cqGc) balance: 8333000000000 mina
2025-08-22T18:57:36.3681136Z 2025-08-22T18:53:06.552882Z INFO Server::run{addr: 0.0.0.0:11004}: warp::server: listening on http://0.0.0.0:11004
2025-08-22T18:57:36.3681565Z waiting for all rust nodes to sync up
2025-08-22T18:57:36.3682648Z 2025-08-22T18:53:07.199159Z INFO mina_core::log::action: time: "1755888784211311364", node_id: 2bF8pb8dwWicAEpuptFvSbLRrx739bdb5uGfRiKwTuRkCBSs6nR, kind: "P2pNetworkKademliaStartBootstrap", summary: "Perform local node's Kademlia bootstrap", key: PeerId(2bF8pb8dwWicAEpuptFvSbLRrx739bdb5uGfRiKwTuRkCBSs6nR)
2025-08-22T18:57:36.3683737Z Non staker total balance: 23234416
2025-08-22T18:57:36.3684002Z [-] no FUZZER conf specified
2025-08-22T18:57:36.3685011Z 2025-08-22T18:53:07.210377Z INFO mina_core::log::action: time: "1755888784211311344", node_id: 2cDEJzT64iZSiqDALyPSwaHtXjxNc96j7wBr9S87QkYiPhrKnAA, kind: "TransitionFrontierGenesisInject", summary: "Inject genesis block into the transition frontier"
2025-08-22T18:57:36.3686319Z 2025-08-22T18:53:07.210399Z INFO mina_core::log::action: time: "1755888784211311345", node_id: 2cDEJzT64iZSiqDALyPSwaHtXjxNc96j7wBr9S87QkYiPhrKnAA, kind: "BlockProducerBestTipUpdate"
2025-08-22T18:57:36.3687639Z 2025-08-22T18:53:07.211869Z INFO mina_core::log::action: time: "1755888784211311346", node_id: 2cDEJzT64iZSiqDALyPSwaHtXjxNc96j7wBr9S87QkYiPhrKnAA, kind: "TransactionPoolBestTipChanged"
2025-08-22T18:57:36.3688784Z 2025-08-22T18:53:07.211879Z INFO node::transaction_pool::transaction_pool_effects: time: "1755888787211879784", kind: "TransactionPoolEffectfulFetchAccounts", summary: "fetching accounts for tx pool"
2025-08-22T18:57:36.3689976Z 2025-08-22T18:53:07.213357Z INFO mina_core::log::action: time: "1755888784211311348", node_id: 2cDEJzT64iZSiqDALyPSwaHtXjxNc96j7wBr9S87QkYiPhrKnAA, kind: "TransactionPoolBestTipChangedWithAccounts"
2025-08-22T18:57:36.3691339Z 2025-08-22T18:53:07.213376Z INFO p2p::p2p_state: time: "1755888787213376388", kind: "P2pState new", summary: "Current node's id: 12D3KooWSCufgHzV4fCwRijfH2k3abrpAJxTKxEvN1FDuRXA2U9x", peer_id_str: "12D3KooWSCufgHzV4fCwRijfH2k3abrpAJxTKxEvN1FDuRXA2U9x"
2025-08-22T18:57:36.3692958Z 2025-08-22T18:53:07.213399Z INFO mina_core::log::action: time: "1755888784211311349", node_id: 2cDEJzT64iZSiqDALyPSwaHtXjxNc96j7wBr9S87QkYiPhrKnAA, kind: "P2pInitializeInitialize", summary: "Initializes p2p layer", chain_id: 89ac2b7463e5f562d55222d76bb7a11c7091ac987b87951698348b8bd809324c
2025-08-22T18:57:36.3693944Z all rust nodes synced up
2025-08-22T18:57:36.3694172Z setting up rust block producer nodes: 3/3
2025-08-22T18:57:36.3694645Z block producer(B62qnLjgW4LAnrxkcdLc7Snb49qx6aP5qsmPsp6ueZN4XPMC621cqGc) stake: 75008333000000000 mina
2025-08-22T18:57:36.3695863Z 2025-08-22T18:53:07.215201Z INFO mina_tree::proofs::circuit_blobs: time: "1755888787215200918", kind: "ledger proofs", "circuit-blobs not found locally, so fetching it...", filename: "3.0.1devnet/step-step-proving-key-blockchain-snark-step-0-55f640777b6486a6fd3fdbc3fcffcc60_gates.json"
2025-08-22T18:57:36.3697959Z 2025-08-22T18:53:09.573184Z INFO mina_tree::proofs::circuit_blobs: time: "1755888789573180649", kind: "ledger proofs", "caching circuit-blobs", path: "/github/home/.openmina/circuit-blobs/3.0.1devnet/step-step-proving-key-blockchain-snark-step-0-55f640777b6486a6fd3fdbc3fcffcc60_gates.json"
2025-08-22T18:57:36.3698929Z
2025-08-22T18:57:36.3718934Z ##[error]panic without a message
2025-08-22T18:57:36.3725535Z thread 'record_replay_block_production' panicked at ledger/src/proofs/provers.rs:73:67:
2025-08-22T18:57:36.3726217Z called `Result::unwrap()` on an `Err` value: Custom { kind: InvalidData, error: Error("expected value", line: 1, column: 1) }
2025-08-22T18:57:36.3726686Z stack backtrace:
2025-08-22T18:57:36.3727450Z 0: 0x5591f4c24d9a - <std::sys::backtrace::BacktraceLock::print::DisplayBacktrace as core::fmt::Display>::fmt::h14b23c1989cbd5c2
2025-08-22T18:57:36.3728025Z 1: 0x5591f4c53483 - core::fmt::write::h346b5eee5ed4d7cc
2025-08-22T18:57:36.3728388Z 2: 0x5591f4c207f3 - std::io::Write::write_fmt::h88e4c2f3647229f3
2025-08-22T18:57:36.3728824Z 3: 0x5591f4c24be2 - std::sys::backtrace::BacktraceLock::print::h48db11f3fd4983ff
2025-08-22T18:57:36.3729330Z 4: 0x5591f4c2621d - std::panicking::default_hook::{{closure}}::h65db976b9c0d8674
2025-08-22T18:57:36.3729769Z 5: 0x5591f4c26027 - std::panicking::default_hook::h6eb3a1192db1ae36
2025-08-22T18:57:36.3730221Z 6: 0x5591f29d89a7 - test::test_main::{{closure}}::h8312a87e3ad01bef
2025-08-22T18:57:36.3730757Z 7: 0x5591f27882bf - record_replay::record_replay_bootstrap::{{closure}}::{{closure}}::h1c9cc9efa1abe81c
2025-08-22T18:57:36.3731285Z 8: 0x5591f4c268c8 - std::panicking::rust_panic_with_hook::h370ee1901241e459
2025-08-22T18:57:36.3731944Z 9: 0x5591f4c2668a - std::panicking::begin_panic_handler::{{closure}}::heabfe92676d6d073
2025-08-22T18:57:36.3732455Z 10: 0x5591f4c25279 - std::sys::backtrace::__rust_end_short_backtrace::h6e22d229d4fdf49e
2025-08-22T18:57:36.3732833Z 11: 0x5591f4c2631c - rust_begin_unwind
2025-08-22T18:57:36.3733163Z 12: 0x5591f265d800 - core::panicking::panic_fmt::hfae270fab21da3e6
2025-08-22T18:57:36.3733558Z 13: 0x5591f265dc46 - core::result::unwrap_failed::h82c78b552479728a
2025-08-22T18:57:36.3733986Z 14: 0x5591f401ac56 - mina_tree::proofs::provers::make_gates::h7f9a8fa14f44843a
2025-08-22T18:57:36.3734538Z 15: 0x5591f417d527 - mina_tree::proofs::provers::prover_makers::get_or_make_block_step_prover::h88e633dadf5d1f67
2025-08-22T18:57:36.3735268Z 16: 0x5591f401b9f3 - mina_tree::proofs::provers::prover_makers::<impl mina_tree::proofs::provers::BlockProver>::make::h2d648f93ae1734be
2025-08-22T18:57:36.3735929Z 17: 0x5591f2ae738a - mina_node_testing::cluster::Cluster::add_rust_node::ha579ae1223934e33
2025-08-22T18:57:36.3736499Z 18: 0x5591f2edf7b4 - mina_node_testing::cluster::runner::ClusterRunner::add_rust_node::h0316abca310c511b
2025-08-22T18:57:36.3737302Z 19: 0x5591f278a2ff - record_replay::record_replay_block_production::{{closure}}::h77db3bbbbb3ca200
2025-08-22T18:57:36.3737889Z 20: 0x5591f27817c6 - tokio::runtime::scheduler::current_thread::Context::enter::h682eb72f98056dff
2025-08-22T18:57:36.3738443Z 21: 0x5591f26e6c9b - tokio::runtime::context::scoped::Scoped<T>::set::hf50be174feea40d6
2025-08-22T18:57:36.3739002Z 22: 0x5591f278146a - tokio::runtime::scheduler::current_thread::CurrentThread::block_on::h49a74df36bf4f4b9
2025-08-22T18:57:36.3739544Z 23: 0x5591f2807682 - tokio::runtime::runtime::Runtime::block_on::hb8a99b1bb6fbe480
2025-08-22T18:57:36.3740005Z 24: 0x5591f276321d - core::ops::function::FnOnce::call_once::hed28f182c5363c07
2025-08-22T18:57:36.3740449Z 25: 0x5591f29dd53b - test::__rust_begin_short_backtrace::h812692078d2a7065
2025-08-22T18:57:36.3740989Z 26: 0x5591f29dceb8 - test::run_test::{{closure}}::h427e5ada87ee2030
2025-08-22T18:57:36.3741440Z 27: 0x5591f29a098a - std::sys::backtrace::__rust_begin_short_backtrace::ha8680ce782bf69f7
2025-08-22T18:57:36.3741949Z 28: 0x5591f29a42e4 - core::ops::function::FnOnce::call_once{{vtable.shim}}::hbc1380969ae25c01
2025-08-22T18:57:36.3742470Z 29: 0x5591f4c2c1cb - std::sys::pal::unix::thread::Thread::new::thread_start::h0feaf4a9a4b2ecde
2025-08-22T18:57:36.3742860Z 30: 0x7faf8f9e9aa4 - <unknown>
2025-08-22T18:57:36.3743102Z 31: 0x7faf8fa76a34 - clone
2025-08-22T18:57:36.3743325Z 32: 0x0 - <unknown>
2025-08-22T18:57:36.3743600Z Flushing recorded actions to disk before shutdown. i=3
2025-08-22T18:57:36.3743941Z Flushing recorded actions to disk before shutdown. i=0
2025-08-22T18:57:36.3744272Z Flushing recorded actions to disk before shutdown. i=1
2025-08-22T18:57:36.3744601Z Flushing recorded actions to disk before shutdown. i=2
2025-08-22T18:57:36.3744810Z
2025-08-22T18:57:36.3744814Z
2025-08-22T18:57:36.3744888Z failures:
2025-08-22T18:57:36.3745071Z record_replay_block_production
2025-08-22T18:57:36.3745233Z
2025-08-22T18:57:36.3745479Z test result: FAILED. 1 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out; finished in 291.37s
2025-08-22T18:57:36.3745804Z
2025-08-22T18:57:36.4413107Z ##[error]Process completed with exit code 101.
2025-08-22T18:57:36.4465018Z Stop and remove container: 67bc7a8cc36f4bc0a6651266fff18134_gcrioo1labs192920minadaemon320beta2939b08dnobledevnet_b6f848
2025-08-22T18:57:36.4470136Z ##[command]/usr/bin/docker rm --force f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3
2025-08-22T18:57:36.5363035Z f1bfb8785e93e6865fbafb3e0b0217a480fa1976eabd6386f501c5930e614aa3
2025-08-22T18:57:36.5392363Z Remove container network: github_network_3e27553cc8cd4602b03c417f2cd7173a
2025-08-22T18:57:36.5396696Z ##[command]/usr/bin/docker network rm github_network_3e27553cc8cd4602b03c417f2cd7173a
2025-08-22T18:57:36.6557498Z github_network_3e27553cc8cd4602b03c417f2cd7173a
2025-08-22T18:57:36.6624146Z Cleaning up orphan processes
https://github.com/o1-labs/mina-rust/actions/runs/17163091085/job/48698273248?pr=1346