Skip to content

Batch client not splitting up large batch submissions #14637

@patrick-schultz

Description

@patrick-schultz

What happened?

Reported by Ben Weisburd here

Version

0.2.132

Relevant log output

File "/Users/weisburd/code/step-pipeline/step_pipeline/batch.py", line 300, in run
    result = self._run_batch_obj()
  File "/Users/weisburd/code/step-pipeline/step_pipeline/batch.py", line 368, in _run_batch_obj
    result = self._batch.run(
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch/batch.py", line 716, in run
    return async_to_blocking(self._async_run(dry_run, verbose, delete_scratch_on_exit, **backend_kwargs))  # type: ignore
  File "/usr/local/lib/python3.9/site-packages/hailtop/utils/utils.py", line 186, in async_to_blocking
    raise exc
  File "/usr/local/lib/python3.9/site-packages/hailtop/utils/utils.py", line 181, in async_to_blocking
    return loop.run_until_complete(task)
  File "/usr/local/lib/python3.9/site-packages/nest_asyncio.py", line 99, in run_until_complete
    return f.result()
  File "/usr/local/Cellar/python@3.9/3.9.18_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/futures.py", line 201, in result
    raise self._exception
  File "/usr/local/Cellar/python@3.9/3.9.18_2/Frameworks/Python.framework/Versions/3.9/lib/python3.9/asyncio/tasks.py", line 256, in __step
    result = coro.send(None)
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch/batch.py", line 747, in _async_run
    run_result = await self._backend._async_run(self, dry_run, verbose, delete_scratch_on_exit, **backend_kwargs)  # pylint: disable=assignment-from-no-return
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch/backend.py", line 901, in _async_run
    await async_batch.submit(disable_progress_bar=disable_progress_bar)
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch_client/aioclient.py", line 1234, in submit
    start_job_group_id, start_job_id = await self._submit(
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch_client/aioclient.py", line 1166, in _submit
    byte_specs_bunches = self._create_bunches(
  File "/usr/local/lib/python3.9/site-packages/hailtop/batch_client/aioclient.py", line 1035, in _create_bunches
    assert n_bytes < max_bunch_bytesize, (
AssertionError: every spec must be less than max_bunch_bytesize, 1048576B, but {"always_run":false,"always_copy_output":false,"job_id":6190,"absolute_parent_ids":[],"in_update_parent_ids":[1073,1069,1082,429

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions