Skip to content

Bump VM type from m5 to m8g#101

Merged
andersy005 merged 2 commits intomainfrom
coiled_vm_type
Jul 21, 2025
Merged

Bump VM type from m5 to m8g#101
andersy005 merged 2 commits intomainfrom
coiled_vm_type

Conversation

@norlandrhagen
Copy link
Collaborator

No description provided.

@andersy005 andersy005 merged commit 391bf5e into main Jul 21, 2025
2 checks passed
@andersy005 andersy005 deleted the coiled_vm_type branch July 21, 2025 06:25
@andersy005
Copy link
Member

it turns out this change breaks dask workloads :(. during my test runs, my dask workers were getting killed. and i found this in the logs:

OSError: /lib/aarch64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/libllvmlite.so)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "/opt/coiled/env/lib/python3.13/site-packages/tornado/ioloop.py", line 758, in _run_callback
    ret = callback()
  File "/opt/coiled/env/lib/python3.13/site-packages/tornado/ioloop.py", line 782, in _discard_future_result
    future.result()
    ~~~~~~~~~~~~~^^
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/worker.py", line 221, in wrapper
    return await method(self, *args, **kwargs)  # type: ignore
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/worker.py", line 1317, in handle_scheduler
    await self.handle_stream(comm)
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/core.py", line 889, in handle_stream
    msgs = await comm.read()
           ^^^^^^^^^^^^^^^^^
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/comm/tcp.py", line 248, in read
    msg = await from_frames(
          ^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/comm/utils.py", line 78, in from_frames
    res = _from_frames()
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/comm/utils.py", line 61, in _from_frames
    return protocol.loads(
           ~~~~~~~~~~~~~~^
        frames, deserialize=deserialize, deserializers=deserializers
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/protocol/core.py", line 175, in loads
    return msgpack.loads(
           ~~~~~~~~~~~~~^
        frames[0], object_hook=_decode_default, use_list=False, **msgpack_opts
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "msgpack/_unpacker.pyx", line 194, in msgpack._cmsgpack.unpackb
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/protocol/core.py", line 172, in _decode_default
    return pickle.loads(sub_header["pickled-obj"], buffers=sub_frames)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/coiled/env/lib/python3.13/site-packages/distributed/protocol/pickle.py", line 95, in loads
    return pickle.loads(x)
           ~~~~~~~~~~~~^^^
  File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/__init__.py", line 355, in <module>
    from dask.array import backends, fft, lib, linalg, ma, overlap, random
  File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/backends.py", line 8, in <module>
    from dask.array.core import Array
  File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/core.py", line 39, in <module>
    from dask.array.chunk_types import is_valid_array_chunk, is_valid_chunk_type
  File "/opt/coiled/env/lib/python3.13/site-packages/dask/array/chunk_types.py", line 124, in <module>
    import sparse
  File "/opt/coiled/env/lib/python3.13/site-packages/sparse/__init__.py", line 48, in <module>
    from sparse.numba_backend import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/coiled/env/lib/python3.13/site-packages/sparse/numba_backend/__init__.py", line 85, in <module>
    from ._common import (
    ...<47 lines>...
    )
  File "/opt/coiled/env/lib/python3.13/site-packages/sparse/numba_backend/_common.py", line 8, in <module>
    import numba
  File "/opt/coiled/env/lib/python3.13/site-packages/numba/__init__.py", line 73, in <module>
    from numba.core import config
  File "/opt/coiled/env/lib/python3.13/site-packages/numba/core/config.py", line 17, in <module>
    import llvmlite.binding as ll
  File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/__init__.py", line 4, in <module>
    from .dylib import *
  File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/dylib.py", line 36, in <module>
    ffi.lib.LLVMPY_AddSymbol.argtypes = [
    ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/ffi.py", line 144, in __getattr__
    cfn = getattr(self._lib, name)
                  ^^^^^^^^^
  File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/ffi.py", line 136, in _lib
    self._load_lib()
    ~~~~~~~~~~~~~~^^
  File "/opt/coiled/env/lib/python3.13/site-packages/llvmlite/binding/ffi.py", line 130, in _load_lib
    raise OSError("Could not find/load shared object file") from e
OSError: Could not find/load shared object file
distributed.nanny - ERROR - Worker process died unexpectedly

andersy005 added a commit that referenced this pull request Aug 5, 2025
* origin/main: (51 commits)
  fix: update integration tests to include env-file and ensure pull request triggers (#130)
  feat: add OCRConfig for improved configuration management (#122)
  Bump prefix-dev/setup-pixi from 0.8.10 to 0.8.14 in the actions group (#125)
  refactor: revamp the CLI and add `LocalBatchManager` for local execution (#113)
  Region aggregation for custom histogram. (#121)
  Tract level aggregations (#119)
  bump icechunk min version (#118)
  add new script for subsetting and rechunking CONUS-404 data (#110)
  Valid rid check (#112)
  Work towards seperating deployment from core library (#111)
  Data flow diagram update (#109)
  County region aggregation (#104)
  Interp less zero (#106)
  revert to intel based vm types (#103)
  standardize tag casing from 'project=OCR' to 'Project=OCR' across multiple scripts (#102)
  Bump VM type from m5 to m8g (#101)
  alt duckdb install (#100)
  adds duckdb to spatial dockerfile (#99)
  merge pixi.toml with pyproject.toml (#97)
  remove empty building risk scores to filter out CA and MX buildings from CONUS (#95)
  ...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants