[Rust] Hang / failure in CI on AMD64 Debian 10 Rust stable test workspace

[asfimport]

As observed first by [~nevi_me] on #9592 (comment)

The Rust CI tests seem to be failing due to a timeout, due to a timeout . For example: https://github.com/apache/arrow/runs/2045186826

Data from [~nevi_me]

running 17 tests
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 958) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 964) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 969) ... ignored
test src/arrow/mod.rs - arrow (line 26) ... ok
test src/column/mod.rs - column (line 38) ... ok
test src/compression.rs - compression (line 25) ... ok
test src/file/mod.rs - file (line 29) ... ok
test src/file/mod.rs - file (line 64) ... ok
test src/file/mod.rs - file (line 81) ... ok
test src/file/statistics.rs - file::statistics (line 23) ... ok
test src/file/properties.rs - file::properties (line 22) ... ok
test src/record/api.rs - record::api::Row::get_column_iter (line 62) ... ok
test src/schema/mod.rs - schema (line 22) ... ok
test src/schema/parser.rs - schema::parser (line 24) ... ok

On my machine, this is the last set of tests that run

test src/schema/types.rs - schema::types::ColumnPath::string (line 540) ... ok
test src/file/statistics.rs - file::statistics (line 23) ... ok
test src/schema/parser.rs - schema::parser (line 24) ... ok
test src/schema/mod.rs - schema (line 22) ... ok
test src/schema/printer.rs - schema::printer (line 23) ... ok

Reporter: Andrew Lamb / @alamb
Assignee: Andrew Lamb / @alamb

Original Issue Attachments:

• Cargo.lock

PRs and other links:

• GitHub Pull Request #9652
• GitHub Pull Request #9653

_{Note: This issue was originally created as ARROW-11896. Please see the migration documentation for further details.}

[asfimport]

Andrew Lamb / @alamb:
I plan to look into this but will not have time until tomorrow

[asfimport]

Andrew Lamb / @alamb:
Datapoint:

When running cargo test --all locally (on macos):

• as of 479dce5, the tests all pass for me locally

• as of bfa99d9 (latest master at time of this writing)

  Here is my Cargo.lock in case that is relevant

[asfimport]

Andrew Lamb / @alamb:
After a cargo update the tests still pass for me locally

[asfimport]

Neville Dipale / @nevi-me:
I've also run cargo test --all on a Windows machine, Linux machine, and Linux via WSL. In all instances, I ran cargo update.
We're either hanging at doc tests, or there's something else happening with the CI environment.

[asfimport]

Andrew Lamb / @alamb:

Timeline

I took a review of commits on https://github.com/apache/arrow/commits/master

4f6adc7: This is the last Rust commit where all CI checks passed
Link to pass CI: https://github.com/apache/arrow/runs/2034747838 (AMD64 Debian 10 Rust stable test workspace; succeeded 3 days ago in 16m 31s)

dec5ab9: First Rust commit on where a CI check failed
Fails in clippy (the tests are working): https://github.com/apache/arrow/runs/2043258080 (which makes me suspicious that that maybe something in the tool chains got upgraded??)

e2c7d95: This is the next Rust commit, CI still failing (on clippy0
Still fails on clippy: https://github.com/apache/arrow/runs/2043298275

bfa99d9: This is the next rust commit on master where we fix clippy
Now we see a new failure on "AMD64 Debian 10 Rust stable test workspace" – https://github.com/apache/arrow/runs/2045186826. However, I can't figure out how to get the logs from that job from github, so I made my own job

[asfimport]

Andrew Lamb / @alamb:
I made a PR to see if I can debug a little more what was going on #9651

It seems to deliver the logs to the UI while I have the page open, so I got to see what actually failed

In this case, https://github.com/apache/arrow/pull/9651/checks?check_run_id=2050214995, it was one of the parquet doctests fals failing in the parquet doctests (perhaps the linker is killed due to memory or something)?

   Doc-tests parquet

running 17 tests
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 958) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 964) ... ignored
test src/arrow/array_reader.rs - arrow::array_reader::StructArrayReader::next_batch (line 969) ... ignored
test src/arrow/mod.rs - arrow (line 26) ... ok
test src/column/mod.rs - column (line 38) ... ok
test src/compression.rs - compression (line 25) ... ok
test src/file/mod.rs - file (line 29) ... ok
test src/file/mod.rs - file (line 64) ... ok
test src/file/mod.rs - file (line 81) ... ok
test src/file/statistics.rs - file::statistics (line 23) ... ok
test src/record/api.rs - record::api::Row::get_column_iter (line 62) ... ok
test src/file/properties.rs - file::properties (line 22) ... ok
test src/schema/parser.rs - schema::parser (line 24) ... FAILED
test src/schema/mod.rs - schema (line 22) ... ok
test src/schema/printer.rs - schema::printer (line 23) ... ok
test src/schema/types.rs - schema::types::ColumnPath::append (line 551) ... ok
test src/schema/types.rs - schema::types::ColumnPath::string (line 540) ... ok

failures:

---- src/schema/parser.rs - schema::parser (line 24) stdout ----
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/rustdoctestfFehFh/rust_out.rust_out.7rcbfp3g-cgu.0.rcgu.o" "-o" "/tmp/rustdoctestfFehFh/rust_out" "/tmp/rustdoctestfFehFh/rust_out.33dyzt1ekirinwy8.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/github/home/target/debug/deps" "-L" "/github/home/target/debug/deps" "-L" "/github/home/target/debug/build/lz4-sys-556dd191705eb188/out" "-L" "/github/home/target/debug/build/zstd-sys-7525c8c4a29a2fce/out" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/github/home/target/debug/deps/libparquet-6ae827fdbd61ccf7.rlib" "/github/home/target/debug/deps/libzstd-0b4d8fff2749365d.rlib" "/github/home/target/debug/deps/libzstd_safe-34b4bc35cbeead2d.rlib" "/github/home/target/debug/deps/libzstd_sys-41be8d07ea408f85.rlib" "/github/home/target/debug/deps/liblz4-3d056ebd790e9b0e.rlib" "/github/home/target/debug/deps/liblz4_sys-5555f68438b62b0c.rlib" "/github/home/target/debug/deps/libbrotli-cff0121c78abaca6.rlib" "/github/home/target/debug/deps/libbrotli_decompressor-22c160a059b30519.rlib" "/github/home/target/debug/deps/liballoc_stdlib-665b901074ea8c57.rlib" "/github/home/target/debug/deps/liballoc_no_stdlib-35e4523e2b39f6f2.rlib" "/github/home/target/debug/deps/libbase64-87f92a73c34bc51e.rlib" "/github/home/target/debug/deps/libflate2-63720bf6fa0d5b3c.rlib" "/github/home/target/debug/deps/libminiz_oxide-9aeb9fa9635cb741.rlib" "/github/home/target/debug/deps/libadler-e5722f94cee63222.rlib" "/github/home/target/debug/deps/libcrc32fast-6d9461d006b880e5.rlib" "/github/home/target/debug/deps/libsnap-f9f5d8c9aa444c9b.rlib" "/github/home/target/debug/deps/libparquet_format-6a6fdcc49807b334.rlib" "/github/home/target/debug/deps/libthrift-24b9e257d44e8657.rlib" "/github/home/target/debug/deps/liblog-c34c04d5bf05faba.rlib" "/github/home/target/debug/deps/libthreadpool-851f110d1cd036f1.rlib" "/github/home/target/debug/deps/libnum_cpus-a0ac008ef7e25e2d.rlib" "/github/home/target/debug/deps/libinteger_encoding-5977e7d9017f90aa.rlib" "/github/home/target/debug/deps/libordered_float-ae74ce3fb47790c2.rlib" "/github/home/target/debug/deps/libarrow-5bd69e39beea4c8e.rlib" "/github/home/target/debug/deps/liblexical_core-3514b31aff84e38f.rlib" "/github/home/target/debug/deps/libarrayvec-40b8d37c66dcf776.rlib" "/github/home/target/debug/deps/libstatic_assertions-b8499fdf315a913a.rlib" "/github/home/target/debug/deps/libprettytable-5c3b36626b239b2d.rlib" "/github/home/target/debug/deps/libencode_unicode-796d275ad52e7181.rlib" "/github/home/target/debug/deps/libatty-54f48db2f374cf0e.rlib" "/github/home/target/debug/deps/libterm-1ee6a6b6844e00d8.rlib" "/github/home/target/debug/deps/libdirs-18629b077931ae20.rlib" "/github/home/target/debug/deps/libunicode_width-561907e3cbff2ad1.rlib" "/github/home/target/debug/deps/librand-6fa160660f0b2d5b.rlib" "/github/home/target/debug/deps/librand_chacha-c4de01ec09cc6b22.rlib" "/github/home/target/debug/deps/libppv_lite86-27ed70786432e789.rlib" "/github/home/target/debug/deps/librand_core-8ccf8f1f7b730b55.rlib" "/github/home/target/debug/deps/libgetrandom-41c18e1478356815.rlib" "/github/home/target/debug/deps/libcfg_if-5027c5fda945db67.rlib" "/github/home/target/debug/deps/libflatbuffers-574169feded7e996.rlib" "/github/home/target/debug/deps/libbitflags-f14a0d5eed0a3c48.rlib" "/github/home/target/debug/deps/libthiserror-0b11c96ca44e4d59.rlib" "/github/home/target/debug/deps/libsmallvec-e310e2c157142037.rlib" "/github/home/target/debug/deps/libcsv-dd29447c1ffcc301.rlib" "/github/home/target/debug/deps/libcsv_core-7eced7c4534046dc.rlib" "/github/home/target/debug/deps/libbstr-e854a0843b82e68b.rlib" "/github/home/target/debug/deps/libregex_automata-532662e9778ce48f.rlib" "/github/home/target/debug/deps/libbyteorder-a6db7e07f13316c2.rlib" "/github/home/target/debug/deps/liblazy_static-4445ce03c8114467.rlib" "/github/home/target/debug/deps/libregex-93024aad4a3baf77.rlib" "/github/home/target/debug/deps/libthread_local-435d51ed841cdbec.rlib" "/github/home/target/debug/deps/libonce_cell-441ca5be97291913.rlib" "/github/home/target/debug/deps/libregex_syntax-a1379624b565cc38.rlib" "/github/home/target/debug/deps/libaho_corasick-6617b386ca598d8a.rlib" "/github/home/target/debug/deps/libmemchr-c9daf75d5b69006b.rlib" "/github/home/target/debug/deps/libserde_json-7e85463fa6e395d1.rlib" "/github/home/target/debug/deps/libryu-c299d28f346c44ca.rlib" "/github/home/target/debug/deps/libitoa-03464e37801631ae.rlib" "/github/home/target/debug/deps/libindexmap-d5ec2ba30ac1e27b.rlib" "/github/home/target/debug/deps/libhashbrown-d049193034b30f6e.rlib" "/github/home/target/debug/deps/libahash-505e78ef4db1f3d7.rlib" "/github/home/target/debug/deps/libserde-caef87319d9bc16e.rlib" "/github/home/target/debug/deps/libhex-fd42229fd072735e.rlib" "/github/home/target/debug/deps/libchrono-7f14bf04429607fe.rlib" "/github/home/target/debug/deps/libtime-e734a2d104f9d0f8.rlib" "/github/home/target/debug/deps/liblibc-1b51117870af79aa.rlib" "/github/home/target/debug/deps/libnum-99facaaa0e43a7e1.rlib" "/github/home/target/debug/deps/libnum_iter-63c7a55e162e47e2.rlib" "/github/home/target/debug/deps/libnum_rational-8a0eb60f91b0f28c.rlib" "/github/home/target/debug/deps/libnum_complex-47a5db1e6352be90.rlib" "/github/home/target/debug/deps/libnum_bigint-ad7e1acea5968d1e.rlib" "/github/home/target/debug/deps/libnum_integer-c919a286b29e87e5.rlib" "/github/home/target/debug/deps/libnum_traits-f4d59dfb8c31f803.rlib" "-Wl,--start-group" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6f77337c1826707d.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-e238540f8faf9927.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-78f8506dd5004913.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-c0e64497deb085aa.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-217a514dc8d845c6.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-65e4f8df4b8bbd73.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-fec1b74170245419.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-e74f867eb2f15322.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-72a374d36f3d0a31.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-cb9206ed96de54c3.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-33ee81149d314805.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-2e8740599b11634b.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-241cf59851357bea.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-edc29aa8c04b480f.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-6a141406945fda5a.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-2408224b4af46d83.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d36087076e1dd756.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc"
  = note: collect2: error: ld returned 1 exit status


error: aborting due to previous error

Couldn't compile the test.

failures:
    src/schema/parser.rs - schema::parser (line 24)

test result: FAILED. 13 passed; 1 failed; 3 ignored; 0 measured; 0 filtered out; finished in 17.10s

error: test failed, to rerun pass '--doc'
Error: Process completed with exit code 101.

But note this fails after 6 mins whereas the failures on master failed after 45 minutes so there is definitely some work to be done

[asfimport]

Andrew Lamb / @alamb:

My next experimental run also failed running the linker in the doc tests but this time in datafusion docs

   Doc-tests datafusion

running 29 tests
test src/dataframe.rs - dataframe::DataFrame::aggregate (line 100) ... FAILED
test src/dataframe.rs - dataframe::DataFrame (line 40) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::collect (line 196) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::collect_partitioned (line 212) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::explain (line 245) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::filter (line 86) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::join (line 152) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::limit (line 123) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::repartition (line 179) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::registry (line 260) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::schema (line 227) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::select (line 72) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::select_columns (line 58) ... FAILED
test src/dataframe.rs - dataframe::DataFrame::sort (line 138) ... FAILED
test src/datasource/csv.rs - datasource::csv (line 24) ... FAILED
test src/execution/context.rs - execution::context::ExecutionContext (line 72) ... FAILED
test src/execution/context.rs - execution::context::ExecutionContext (line 88) ... FAILED
test src/lib.rs - (line 35) ... FAILED
test src/lib.rs - (line 60) ... FAILED
test src/logical_plan/display.rs - logical_plan::display::display_schema (line 90) ... ok
test src/logical_plan/expr.rs - logical_plan::expr::Expr (line 56) ... FAILED
test src/logical_plan/expr.rs - logical_plan::expr::Expr (line 49) ... ok
test src/logical_plan/expr.rs - logical_plan::expr::Expr (line 69) ... FAILED
test src/logical_plan/expr.rs - logical_plan::expr::Expr::sort (line 459) ... ok
test src/logical_plan/plan.rs - logical_plan::plan::LogicalPlan::display_graphviz (line 502) ... FAILED
test src/logical_plan/plan.rs - logical_plan::plan::LogicalPlan::display (line 561) ... ok
test src/logical_plan/plan.rs - logical_plan::plan::LogicalPlan::display_indent_schema (line 462) ... FAILED
test src/prelude.rs - prelude (line 24) ... ok
test src/logical_plan/plan.rs - logical_plan::plan::LogicalPlan::display_indent (line 421) ... ok

failures:

---- src/dataframe.rs - dataframe::DataFrame::aggregate (line 100) stdout ----
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/rustdoctestspSAFP/rust_out.rust_out.7rcbfp3g-cgu.0.rcgu.o" "-o" "/tmp/rustdoctestspSAFP/rust_out" "/tmp/rustdoctestspSAFP/rust_out.33dyzt1ekirinwy8.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/github/home/target/debug/deps" "-L" "/github/home/target/debug/deps" "-L" "/github/home/target/debug/build/lz4-sys-556dd191705eb188/out" "-L" "/github/home/target/debug/build/zstd-sys-7525c8c4a29a2fce/out" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/github/home/target/debug/deps/libdatafusion-750b7235538956cc.rlib" "/github/home/target/debug/deps/libunicode_segmentation-b121d2f101ca84a9.rlib" "/github/home/target/debug/deps/libtokio_stream-726d2c35cab16fd5.rlib" "/github/home/target/debug/deps/libordered_float-65d76fe97cbf8cb1.rlib" "/github/home/target/debug/deps/libahash-80bffc56238765fe.rlib" "/github/home/target/debug/deps/libgetrandom-9fa2c570f1676634.rlib" "/github/home/target/debug/deps/libsha2-e8e2dec6afd269a2.rlib" "/github/home/target/debug/deps/libcpuid_bool-bf42cd064aa1c533.rlib" "/github/home/target/debug/deps/libmd5-598da4ca8a369b0c.rlib" "/github/home/target/debug/deps/libopaque_debug-bd597894d6d7e307.rlib" "/github/home/target/debug/deps/libblock_buffer-55d31eb95083ed64.rlib" "/github/home/target/debug/deps/libdigest-6e452b9ac411a1cf.rlib" "/github/home/target/debug/deps/libgeneric_array-9c349825dff5b1fa.rlib" "/github/home/target/debug/deps/libtypenum-6e04b22406d9ec0d.rlib" "/github/home/target/debug/deps/libtokio-620dbf4ef2870414.rlib" "/github/home/target/debug/deps/libbytes-dbb60275a9c48d19.rlib" "/github/home/target/debug/deps/libmio-c526b4518634eb00.rlib" "/github/home/target/debug/deps/libparquet-6ae827fdbd61ccf7.rlib" "/github/home/target/debug/deps/libzstd-0b4d8fff2749365d.rlib" "/github/home/target/debug/deps/libzstd_safe-34b4bc35cbeead2d.rlib" "/github/home/target/debug/deps/libzstd_sys-41be8d07ea408f85.rlib" "/github/home/target/debug/deps/liblz4-3d056ebd790e9b0e.rlib" "/github/home/target/debug/deps/liblz4_sys-5555f68438b62b0c.rlib" "/github/home/target/debug/deps/libbrotli-cff0121c78abaca6.rlib" "/github/home/target/debug/deps/libbrotli_decompressor-22c160a059b30519.rlib" "/github/home/target/debug/deps/liballoc_stdlib-665b901074ea8c57.rlib" "/github/home/target/debug/deps/liballoc_no_stdlib-35e4523e2b39f6f2.rlib" "/github/home/target/debug/deps/libbase64-87f92a73c34bc51e.rlib" "/github/home/target/debug/deps/libflate2-63720bf6fa0d5b3c.rlib" "/github/home/target/debug/deps/libminiz_oxide-9aeb9fa9635cb741.rlib" "/github/home/target/debug/deps/libadler-e5722f94cee63222.rlib" "/github/home/target/debug/deps/libcrc32fast-6d9461d006b880e5.rlib" "/github/home/target/debug/deps/libsnap-f9f5d8c9aa444c9b.rlib" "/github/home/target/debug/deps/libparquet_format-6a6fdcc49807b334.rlib" "/github/home/target/debug/deps/libthrift-24b9e257d44e8657.rlib" "/github/home/target/debug/deps/libthreadpool-851f110d1cd036f1.rlib" "/github/home/target/debug/deps/libnum_cpus-a0ac008ef7e25e2d.rlib" "/github/home/target/debug/deps/libinteger_encoding-5977e7d9017f90aa.rlib" "/github/home/target/debug/deps/libordered_float-ae74ce3fb47790c2.rlib" "/github/home/target/debug/deps/libfutures-0e86d0a9ce18e3ba.rlib" "/github/home/target/debug/deps/libfutures_executor-ed0229d1d264d608.rlib" "/github/home/target/debug/deps/libfutures_util-32aceb3df493bcaf.rlib" "/github/home/target/debug/deps/libfutures_io-19f03005b76a692b.rlib" "/github/home/target/debug/deps/libslab-d3011466083081da.rlib" "/github/home/target/debug/deps/libfutures_channel-21bc467eec455854.rlib" "/github/home/target/debug/deps/libpin_project_lite-1090cf562b4a2ee8.rlib" "/github/home/target/debug/deps/libfutures_sink-9cc3ef9ae6f8f102.rlib" "/github/home/target/debug/deps/libfutures_task-5fbbabe6e4980072.rlib" "/github/home/target/debug/deps/libpin_utils-12d03d6fcdca3843.rlib" "/github/home/target/debug/deps/libfutures_core-6ba47a20838b0d73.rlib" "/github/home/target/debug/deps/libsqlparser-918d5c8e41ac8048.rlib" "/github/home/target/debug/deps/liblog-c34c04d5bf05faba.rlib" "/github/home/target/debug/deps/libarrow-5bd69e39beea4c8e.rlib" "/github/home/target/debug/deps/liblexical_core-3514b31aff84e38f.rlib" "/github/home/target/debug/deps/libarrayvec-40b8d37c66dcf776.rlib" "/github/home/target/debug/deps/libstatic_assertions-b8499fdf315a913a.rlib" "/github/home/target/debug/deps/libprettytable-5c3b36626b239b2d.rlib" "/github/home/target/debug/deps/libencode_unicode-796d275ad52e7181.rlib" "/github/home/target/debug/deps/libatty-54f48db2f374cf0e.rlib" "/github/home/target/debug/deps/libterm-1ee6a6b6844e00d8.rlib" "/github/home/target/debug/deps/libdirs-18629b077931ae20.rlib" "/github/home/target/debug/deps/libunicode_width-561907e3cbff2ad1.rlib" "/github/home/target/debug/deps/librand-6fa160660f0b2d5b.rlib" "/github/home/target/debug/deps/librand_chacha-c4de01ec09cc6b22.rlib" "/github/home/target/debug/deps/libppv_lite86-27ed70786432e789.rlib" "/github/home/target/debug/deps/librand_core-8ccf8f1f7b730b55.rlib" "/github/home/target/debug/deps/libgetrandom-41c18e1478356815.rlib" "/github/home/target/debug/deps/libcfg_if-5027c5fda945db67.rlib" "/github/home/target/debug/deps/libflatbuffers-574169feded7e996.rlib" "/github/home/target/debug/deps/libbitflags-f14a0d5eed0a3c48.rlib" "/github/home/target/debug/deps/libthiserror-0b11c96ca44e4d59.rlib" "/github/home/target/debug/deps/libsmallvec-e310e2c157142037.rlib" "/github/home/target/debug/deps/libcsv-dd29447c1ffcc301.rlib" "/github/home/target/debug/deps/libcsv_core-7eced7c4534046dc.rlib" "/github/home/target/debug/deps/libbstr-e854a0843b82e68b.rlib" "/github/home/target/debug/deps/libregex_automata-532662e9778ce48f.rlib" "/github/home/target/debug/deps/libbyteorder-a6db7e07f13316c2.rlib" "/github/home/target/debug/deps/liblazy_static-4445ce03c8114467.rlib" "/github/home/target/debug/deps/libregex-93024aad4a3baf77.rlib" "/github/home/target/debug/deps/libthread_local-435d51ed841cdbec.rlib" "/github/home/target/debug/deps/libonce_cell-441ca5be97291913.rlib" "/github/home/target/debug/deps/libregex_syntax-a1379624b565cc38.rlib" "/github/home/target/debug/deps/libaho_corasick-6617b386ca598d8a.rlib" "/github/home/target/debug/deps/libmemchr-c9daf75d5b69006b.rlib" "/github/home/target/debug/deps/libserde_json-7e85463fa6e395d1.rlib" "/github/home/target/debug/deps/libryu-c299d28f346c44ca.rlib" "/github/home/target/debug/deps/libitoa-03464e37801631ae.rlib" "/github/home/target/debug/deps/libindexmap-d5ec2ba30ac1e27b.rlib" "/github/home/target/debug/deps/libhashbrown-d049193034b30f6e.rlib" "/github/home/target/debug/deps/libahash-505e78ef4db1f3d7.rlib" "/github/home/target/debug/deps/libserde-caef87319d9bc16e.rlib" "/github/home/target/debug/deps/libhex-fd42229fd072735e.rlib" "/github/home/target/debug/deps/libchrono-7f14bf04429607fe.rlib" "/github/home/target/debug/deps/libtime-e734a2d104f9d0f8.rlib" "/github/home/target/debug/deps/liblibc-1b51117870af79aa.rlib" "/github/home/target/debug/deps/libnum-99facaaa0e43a7e1.rlib" "/github/home/target/debug/deps/libnum_iter-63c7a55e162e47e2.rlib" "/github/home/target/debug/deps/libnum_rational-8a0eb60f91b0f28c.rlib" "/github/home/target/debug/deps/libnum_complex-47a5db1e6352be90.rlib" "/github/home/target/debug/deps/libnum_bigint-ad7e1acea5968d1e.rlib" "/github/home/target/debug/deps/libnum_integer-c919a286b29e87e5.rlib" "/github/home/target/debug/deps/libnum_traits-f4d59dfb8c31f803.rlib" "-Wl,--start-group" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6f77337c1826707d.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-e238540f8faf9927.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-78f8506dd5004913.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-c0e64497deb085aa.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-217a514dc8d845c6.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-65e4f8df4b8bbd73.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-fec1b74170245419.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-e74f867eb2f15322.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-72a374d36f3d0a31.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-cb9206ed96de54c3.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-33ee81149d314805.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-2e8740599b11634b.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-241cf59851357bea.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-edc29aa8c04b480f.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-6a141406945fda5a.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-2408224b4af46d83.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d36087076e1dd756.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc"
  = note: collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

Couldn't compile the test.
---- src/dataframe.rs - dataframe::DataFrame (line 40) stdout ----
error: linking with `cc` failed: exit code: 1
  |
  = note: "cc" "-Wl,--as-needed" "-Wl,-z,noexecstack" "-m64" "-Wl,--eh-frame-hdr" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "/tmp/rustdoctestzwBYiv/rust_out.rust_out.7rcbfp3g-cgu.0.rcgu.o" "-o" "/tmp/rustdoctestzwBYiv/rust_out" "/tmp/rustdoctestzwBYiv/rust_out.33dyzt1ekirinwy8.rcgu.o" "-Wl,--gc-sections" "-pie" "-Wl,-zrelro" "-Wl,-znow" "-nodefaultlibs" "-L" "/github/home/target/debug/deps" "-L" "/github/home/target/debug/deps" "-L" "/github/home/target/debug/build/lz4-sys-556dd191705eb188/out" "-L" "/github/home/target/debug/build/zstd-sys-7525c8c4a29a2fce/out" "-L" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "/github/home/target/debug/deps/libdatafusion-750b7235538956cc.rlib" "/github/home/target/debug/deps/libunicode_segmentation-b121d2f101ca84a9.rlib" "/github/home/target/debug/deps/libtokio_stream-726d2c35cab16fd5.rlib" "/github/home/target/debug/deps/libordered_float-65d76fe97cbf8cb1.rlib" "/github/home/target/debug/deps/libahash-80bffc56238765fe.rlib" "/github/home/target/debug/deps/libgetrandom-9fa2c570f1676634.rlib" "/github/home/target/debug/deps/libsha2-e8e2dec6afd269a2.rlib" "/github/home/target/debug/deps/libcpuid_bool-bf42cd064aa1c533.rlib" "/github/home/target/debug/deps/libmd5-598da4ca8a369b0c.rlib" "/github/home/target/debug/deps/libopaque_debug-bd597894d6d7e307.rlib" "/github/home/target/debug/deps/libblock_buffer-55d31eb95083ed64.rlib" "/github/home/target/debug/deps/libdigest-6e452b9ac411a1cf.rlib" "/github/home/target/debug/deps/libgeneric_array-9c349825dff5b1fa.rlib" "/github/home/target/debug/deps/libtypenum-6e04b22406d9ec0d.rlib" "/github/home/target/debug/deps/libtokio-620dbf4ef2870414.rlib" "/github/home/target/debug/deps/libbytes-dbb60275a9c48d19.rlib" "/github/home/target/debug/deps/libmio-c526b4518634eb00.rlib" "/github/home/target/debug/deps/libparquet-6ae827fdbd61ccf7.rlib" "/github/home/target/debug/deps/libzstd-0b4d8fff2749365d.rlib" "/github/home/target/debug/deps/libzstd_safe-34b4bc35cbeead2d.rlib" "/github/home/target/debug/deps/libzstd_sys-41be8d07ea408f85.rlib" "/github/home/target/debug/deps/liblz4-3d056ebd790e9b0e.rlib" "/github/home/target/debug/deps/liblz4_sys-5555f68438b62b0c.rlib" "/github/home/target/debug/deps/libbrotli-cff0121c78abaca6.rlib" "/github/home/target/debug/deps/libbrotli_decompressor-22c160a059b30519.rlib" "/github/home/target/debug/deps/liballoc_stdlib-665b901074ea8c57.rlib" "/github/home/target/debug/deps/liballoc_no_stdlib-35e4523e2b39f6f2.rlib" "/github/home/target/debug/deps/libbase64-87f92a73c34bc51e.rlib" "/github/home/target/debug/deps/libflate2-63720bf6fa0d5b3c.rlib" "/github/home/target/debug/deps/libminiz_oxide-9aeb9fa9635cb741.rlib" "/github/home/target/debug/deps/libadler-e5722f94cee63222.rlib" "/github/home/target/debug/deps/libcrc32fast-6d9461d006b880e5.rlib" "/github/home/target/debug/deps/libsnap-f9f5d8c9aa444c9b.rlib" "/github/home/target/debug/deps/libparquet_format-6a6fdcc49807b334.rlib" "/github/home/target/debug/deps/libthrift-24b9e257d44e8657.rlib" "/github/home/target/debug/deps/libthreadpool-851f110d1cd036f1.rlib" "/github/home/target/debug/deps/libnum_cpus-a0ac008ef7e25e2d.rlib" "/github/home/target/debug/deps/libinteger_encoding-5977e7d9017f90aa.rlib" "/github/home/target/debug/deps/libordered_float-ae74ce3fb47790c2.rlib" "/github/home/target/debug/deps/libfutures-0e86d0a9ce18e3ba.rlib" "/github/home/target/debug/deps/libfutures_executor-ed0229d1d264d608.rlib" "/github/home/target/debug/deps/libfutures_util-32aceb3df493bcaf.rlib" "/github/home/target/debug/deps/libfutures_io-19f03005b76a692b.rlib" "/github/home/target/debug/deps/libslab-d3011466083081da.rlib" "/github/home/target/debug/deps/libfutures_channel-21bc467eec455854.rlib" "/github/home/target/debug/deps/libpin_project_lite-1090cf562b4a2ee8.rlib" "/github/home/target/debug/deps/libfutures_sink-9cc3ef9ae6f8f102.rlib" "/github/home/target/debug/deps/libfutures_task-5fbbabe6e4980072.rlib" "/github/home/target/debug/deps/libpin_utils-12d03d6fcdca3843.rlib" "/github/home/target/debug/deps/libfutures_core-6ba47a20838b0d73.rlib" "/github/home/target/debug/deps/libsqlparser-918d5c8e41ac8048.rlib" "/github/home/target/debug/deps/liblog-c34c04d5bf05faba.rlib" "/github/home/target/debug/deps/libarrow-5bd69e39beea4c8e.rlib" "/github/home/target/debug/deps/liblexical_core-3514b31aff84e38f.rlib" "/github/home/target/debug/deps/libarrayvec-40b8d37c66dcf776.rlib" "/github/home/target/debug/deps/libstatic_assertions-b8499fdf315a913a.rlib" "/github/home/target/debug/deps/libprettytable-5c3b36626b239b2d.rlib" "/github/home/target/debug/deps/libencode_unicode-796d275ad52e7181.rlib" "/github/home/target/debug/deps/libatty-54f48db2f374cf0e.rlib" "/github/home/target/debug/deps/libterm-1ee6a6b6844e00d8.rlib" "/github/home/target/debug/deps/libdirs-18629b077931ae20.rlib" "/github/home/target/debug/deps/libunicode_width-561907e3cbff2ad1.rlib" "/github/home/target/debug/deps/librand-6fa160660f0b2d5b.rlib" "/github/home/target/debug/deps/librand_chacha-c4de01ec09cc6b22.rlib" "/github/home/target/debug/deps/libppv_lite86-27ed70786432e789.rlib" "/github/home/target/debug/deps/librand_core-8ccf8f1f7b730b55.rlib" "/github/home/target/debug/deps/libgetrandom-41c18e1478356815.rlib" "/github/home/target/debug/deps/libcfg_if-5027c5fda945db67.rlib" "/github/home/target/debug/deps/libflatbuffers-574169feded7e996.rlib" "/github/home/target/debug/deps/libbitflags-f14a0d5eed0a3c48.rlib" "/github/home/target/debug/deps/libthiserror-0b11c96ca44e4d59.rlib" "/github/home/target/debug/deps/libsmallvec-e310e2c157142037.rlib" "/github/home/target/debug/deps/libcsv-dd29447c1ffcc301.rlib" "/github/home/target/debug/deps/libcsv_core-7eced7c4534046dc.rlib" "/github/home/target/debug/deps/libbstr-e854a0843b82e68b.rlib" "/github/home/target/debug/deps/libregex_automata-532662e9778ce48f.rlib" "/github/home/target/debug/deps/libbyteorder-a6db7e07f13316c2.rlib" "/github/home/target/debug/deps/liblazy_static-4445ce03c8114467.rlib" "/github/home/target/debug/deps/libregex-93024aad4a3baf77.rlib" "/github/home/target/debug/deps/libthread_local-435d51ed841cdbec.rlib" "/github/home/target/debug/deps/libonce_cell-441ca5be97291913.rlib" "/github/home/target/debug/deps/libregex_syntax-a1379624b565cc38.rlib" "/github/home/target/debug/deps/libaho_corasick-6617b386ca598d8a.rlib" "/github/home/target/debug/deps/libmemchr-c9daf75d5b69006b.rlib" "/github/home/target/debug/deps/libserde_json-7e85463fa6e395d1.rlib" "/github/home/target/debug/deps/libryu-c299d28f346c44ca.rlib" "/github/home/target/debug/deps/libitoa-03464e37801631ae.rlib" "/github/home/target/debug/deps/libindexmap-d5ec2ba30ac1e27b.rlib" "/github/home/target/debug/deps/libhashbrown-d049193034b30f6e.rlib" "/github/home/target/debug/deps/libahash-505e78ef4db1f3d7.rlib" "/github/home/target/debug/deps/libserde-caef87319d9bc16e.rlib" "/github/home/target/debug/deps/libhex-fd42229fd072735e.rlib" "/github/home/target/debug/deps/libchrono-7f14bf04429607fe.rlib" "/github/home/target/debug/deps/libtime-e734a2d104f9d0f8.rlib" "/github/home/target/debug/deps/liblibc-1b51117870af79aa.rlib" "/github/home/target/debug/deps/libnum-99facaaa0e43a7e1.rlib" "/github/home/target/debug/deps/libnum_iter-63c7a55e162e47e2.rlib" "/github/home/target/debug/deps/libnum_rational-8a0eb60f91b0f28c.rlib" "/github/home/target/debug/deps/libnum_complex-47a5db1e6352be90.rlib" "/github/home/target/debug/deps/libnum_bigint-ad7e1acea5968d1e.rlib" "/github/home/target/debug/deps/libnum_integer-c919a286b29e87e5.rlib" "/github/home/target/debug/deps/libnum_traits-f4d59dfb8c31f803.rlib" "-Wl,--start-group" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libstd-6f77337c1826707d.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libpanic_unwind-e238540f8faf9927.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libminiz_oxide-78f8506dd5004913.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libadler-c0e64497deb085aa.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libobject-217a514dc8d845c6.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libaddr2line-65e4f8df4b8bbd73.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libgimli-fec1b74170245419.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_demangle-e74f867eb2f15322.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libhashbrown-72a374d36f3d0a31.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_alloc-cb9206ed96de54c3.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libunwind-33ee81149d314805.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcfg_if-2e8740599b11634b.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liblibc-241cf59851357bea.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc-edc29aa8c04b480f.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/librustc_std_workspace_core-6a141406945fda5a.rlib" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcore-2408224b4af46d83.rlib" "-Wl,--end-group" "/usr/local/rustup/toolchains/stable-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/lib/libcompiler_builtins-d36087076e1dd756.rlib" "-Wl,-Bdynamic" "-lgcc_s" "-lutil" "-lrt" "-lpthread" "-lm" "-ldl" "-lc"
  = note: collect2: error: ld returned 1 exit status
          

error: aborting due to previous error

[asfimport]

Andrew Lamb / @alamb:
Hmm, I also tried running the tests single threaded, but I am still getting linker errors here: this: #9652

[asfimport]

Andrew Lamb / @alamb:
My current theory is that the final link command is consuming right about the amout of memory the github builders have available and thus is being OOM killed. I am investigating ways to reduce the memory used

[asfimport]

Andrew Lamb / @alamb:
Issue resolved by pull request 9653
#9653