Merged
Conversation
Member
|
it turns out this change breaks dask workloads :(. during my test runs, my dask workers were getting killed. and i found this in the logs: OSError: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/libllvmlite.so)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/coiled/env/lib/python3.13/site-packages/tornado/ioloop.py", line 758, in _run_callback
ret = callback()
File "/opt/coiled/env/lib/python3.13/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
future.result()
~~~~~~~~~~~~~^^
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/worker.py", line 221, in wrapper
return await method(self, *args, **kwargs) # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/worker.py", line 1317, in handle_scheduler
await self.handle_stream(comm)
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/core.py", line 889, in handle_stream
msgs = await comm.read()
^^^^^^^^^^^^^^^^^
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/comm/tcp.py", line 248, in read
msg = await from_frames(
^^^^^^^^^^^^^^^^^^
...<4 lines>...
)
^
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/comm/utils.py", line 78, in from_frames
res = _from_frames()
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/comm/utils.py", line 61, in _from_frames
return protocol.loads(
~~~~~~~~~~~~~~^
frames, deserialize=deserialize, deserializers=deserializers
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/protocol/core.py", line 175, in loads
return msgpack.loads(
~~~~~~~~~~~~~^
frames[0], object_hook=_decode_default, use_list=False, **msgpack_opts
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/protocol/core.py", line 172, in _decode_default
return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/coiled/env/lib/python3.13/site-packages/distributed/protocol/pickle.py", line 95, in loads
return pickle.loads(x)
~~~~~~~~~~~~^^^
File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/__init__.py", line 355, in <module>
from dask.array import backends, fft, lib, linalg, ma, overlap, random
File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/backends.py", line 8, in <module>
from dask.array.core import Array
File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/core.py", line 39, in <module>
from dask.array.chunk_types import is_valid_array_chunk, is_valid_chunk_type
File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/chunk_types.py", line 124, in <module>
import sparse
File "/opt/coiled/env/lib/python3.13/site-packages/sparse/__init__.py", line 48, in <module>
from sparse.numba_backend import * # noqa: F403
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/coiled/env/lib/python3.13/site-packages/sparse/numba_backend/__init__.py", line 85, in <module>
from ._common import (
...<47 lines>...
)
File "/opt/coiled/env/lib/python3.13/site-packages/sparse/numba_backend/_common.py", line 8, in <module>
import numba
File "/opt/coiled/env/lib/python3.13/site-packages/numba/__init__.py", line 73, in <module>
from numba.core import config
File "/opt/coiled/env/lib/python3.13/site-packages/numba/core/config.py", line 17, in <module>
import llvmlite.binding as ll
File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/__init__.py", line 4, in <module>
from .dylib import *
File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/dylib.py", line 36, in <module>
ffi.lib.LLVMPY_AddSymbol.argtypes = [
^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/ffi.py", line 144, in __getattr__
cfn = getattr(self._lib, name)
^^^^^^^^^
File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/ffi.py", line 136, in _lib
self._load_lib()
~~~~~~~~~~~~~~^^
File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/ffi.py", line 130, in _load_lib
raise OSError("Could not find/load shared object file") from e
OSError: Could not find/load shared object file
distributed.nanny - ERROR - Worker process died unexpectedly |
andersy005
added a commit
that referenced
this pull request
Aug 5, 2025
* origin/main: (51 commits) fix: update integration tests to include env-file and ensure pull request triggers (#130) feat: add OCRConfig for improved configuration management (#122) Bump prefix-dev/setup-pixi from 0.8.10 to 0.8.14 in the actions group (#125) refactor: revamp the CLI and add `LocalBatchManager` for local execution (#113) Region aggregation for custom histogram. (#121) Tract level aggregations (#119) bump icechunk min version (#118) add new script for subsetting and rechunking CONUS-404 data (#110) Valid rid check (#112) Work towards seperating deployment from core library (#111) Data flow diagram update (#109) County region aggregation (#104) Interp less zero (#106) revert to intel based vm types (#103) standardize tag casing from 'project=OCR' to 'Project=OCR' across multiple scripts (#102) Bump VM type from m5 to m8g (#101) alt duckdb install (#100) adds duckdb to spatial dockerfile (#99) merge pixi.toml with pyproject.toml (#97) remove empty building risk scores to filter out CA and MX buildings from CONUS (#95) ...
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.