feat(trim): explain to users that TRIM expects measurements that always produce the observed property it tracks #598
Description
Is your feature request related to a problem? Please describe. A clear and
The current version of TRIM assumes that during its "iterative modeling" phase all measurements produce the observed property that TRIM trackes. If a measurement does not, then TRIM exits with an exception that does not explain why the iterative modeling phase failed.
raise ValueError("pd.DataFrame input must contain exactly one row")
Describe the solution you'd like. A clear and concise description of what
you want to happen.
I'd like:
- The TRIM docs to list its limitations
- The TRIM code to raise an exception that users can understand
Describe alternatives you've considered. A clear and concise description of
any alternative solutions or features you've considered.
Update the TRIM code to support measurements during its iterative modeling which do not produce the observed property it tracks.
Unfortunately, this will likely require significant effort.
Additional context. Add any other context or screenshots about the feature
request here.
The stacktrace and logs leading up to the exception
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:49,609 INFO AsyncIO Thread: default space_manager : shutdown : Received shutdown - waiting on measurement queue observation to complete
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:49,609 INFO AsyncIO Thread: default space_manager : shutdown : Not complete - waiting 10 secs
(RandomWalk pid=97669) 2026-02-20 09:51:49,582 ERROR AsyncIO Thread: default trim.utils.logging_utils: log_after_split_common_and_diff: 0 point(s) sampled (expected 1), saving data in debug_output
(RandomWalk pid=97669) Continuous Batching: EXPERIMENT COMPLETION. Received finished notification for experiment in measurement request in group 0: request-38b9d850-experiment-finetune_full_benchmark-v1.0.0-entities-model_name.smollm2-135m-number_gpus.0-model_max_length.8192-batch_size.8
(unk)-requester-randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399-time-2026-02-20 09:50:16.462189+00:00
(RandomWalk pid=97669) Continuous Batching: EXPERIMENT FAILURE. Experiment request 38b9d850 with measurement request index 0 failed
(RandomWalk pid=97669) Continuous Batching: EXPERIMENT RETRY. Max retries 0 reached for request 38b9d850
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:54,595 INFO AsyncIO Thread: default space_manager : _monitor_updates_private: Did not get a new measurement after 5 secs - will continue waiting
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:54,596 INFO AsyncIO Thread: default space_manager : _monitor_updates_private: Awaiting 1 sent updates
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:54,596 INFO AsyncIO Thread: default space_manager : _monitor_updates_private: All updates processed
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:54,596 INFO AsyncIO Thread: default space_manager : _monitor_updates_private: Measurement queue observation complete - notifying subscribers
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:54,596 INFO AsyncIO Thread: default space_manager : _monitor_updates_private: Notifying subscriber
(RandomWalk pid=97669) 2026-02-20 09:51:54,598 INFO AsyncIO Thread: default RandomWalk : onCompleted : Completed
(DiscoverySpaceManager pid=97673) 2026-02-20 09:51:59,610 INFO AsyncIO Thread: default space_manager : shutdown : Shutdown complete
2026-02-20 09:51:59,718 INFO MainThread orchestrate_core: _run_operation_harness: Sending final status for operation None to metastore
=========== Operation Details ============
Space ID: space-dafd6d-941b7b
Sample Store ID: 941b7b
Operation:
config:
actuatorConfigurationIdentifiers:
- actuatorconfiguration-SFTTrainer-78d001a9
metadata: {}
operation:
module:
operationType: characterize
operatorName: trim
parameters:
autoGluonArgs:
fitArgs:
excluded_model_types:
- GBM
presets: medium
time_limit: 10
tabularPredictorArgs:
problem_type: regression
batchSize: 1
iterationSize: 5
noPriorsParameters:
samples: 5
targetOutput: dataset_tokens_per_second_per_gpu
outputDirectory: trim_models
samplingBudget:
maxPoints: 12
minPoints: 6
stoppingCriterion:
enabled: true
meanThreshold: 0.9
stdThreshold: 0.75
targetOutput: dataset_tokens_per_second_per_gpu
taskId: null
taskValue: null
spaces:
- space-dafd6d-941b7b
created: '2026-02-20T09:50:09.481462Z'
identifier: operation-trim-v0.1-9e3ab70d
kind: operation
metadata: {}
operationType: characterize
operatorIdentifier: trim-v0.1
status:
- event: created
recorded_at: '2026-02-20T09:50:09.481465Z'
- event: added
recorded_at: '2026-02-20T09:50:09.481790Z'
- event: started
recorded_at: '2026-02-20T09:50:09.485564Z'
- event: updated
recorded_at: '2026-02-20T09:50:09.485573Z'
- event: finished
exit_state: error
message: "Operation exited due to the following error: ('Error raised while executing\
\ operation randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399', OperationResource(version='v1',\
\ kind=<CoreResourceKinds.OPERATION: 'operation'>, created=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 848179, tzinfo=datetime.timezone.utc), config=DiscoveryOperationResourceConfiguration(operation=DiscoveryOperationConfiguration(module=OperatorModuleConf(moduleType=<ModuleTypeEnum.OPERATION:\
\ 'operation'>, moduleName='orchestrator.modules.operators.randomwalk', modulePath='.',\
\ moduleClass='RandomWalk', moduleFunction=None), parameters={'samplerConfig':\
\ {'module': {'moduleType': <ModuleTypeEnum.SAMPLER: 'sampler'>, 'moduleName':\
\ 'trim.trim_sampler', 'modulePath': '.', 'moduleClass': 'TrimSampleSelector',\
\ 'moduleFunction': None}, 'parameters': {'autoGluonArgs': {'tabularPredictorArgs':\
\ {'problem_type': 'regression', 'path': 'trim_models'}, 'fitArgs': {'time_limit':\
\ 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}}, 'finalModelAutoGluonArgs':\
\ {'tabularPredictorArgs': {'problem_type': 'regression', 'path': 'trim_models'},\
\ 'fitArgs': {'time_limit': 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}},\
\ 'targetOutput': 'dataset_tokens_per_second_per_gpu', 'outputDirectory': 'trim_models',\
\ 'debugDirectory': 'debug_output', 'iterationSize': 5, 'holdoutSize': 5, 'samplingBudget':\
\ {'minPoints': 6, 'maxPoints': 12}, 'stoppingCriterion': {'enabled': True, 'meanThreshold':\
\ 0.9, 'stdThreshold': 0.75}, 'noPriorParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu',\
\ 'samples': 6, 'batchSize': 1, 'sampling_strategy': 'clhs'}, 'batchSize': 1,\
\ 'noPriorsParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu',\
\ 'samples': 5}, 'taskId': None, 'taskValue': None}}, 'numberEntities': 12, 'batchSize':\
\ 1, 'singleMeasurement': True, 'maxRetries': 0, 'filter': {'filterMode': <FilterModeEnum.noFilter:\
\ 'noFilter'>}}), metadata=ConfigurationMetadata(name=None, description=None,\
\ labels=None, completed operation='Iterative Modeling Operation'), actuatorConfigurationIdentifiers=['actuatorconfiguration-SFTTrainer-78d001a9'],\
\ spaces=['space-dafd6d-941b7b']), identifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399',\
\ status=[OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 848182, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.CREATED:\
\ 'created'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 848482, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.ADDED:\
\ 'added'>), OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 850608, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.STARTED:\
\ 'started'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 850613, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED:\
\ 'updated'>), OperationResourceStatus(message='Operation exited due to the following\
\ error from a Ray Task: \\x1b[36mray::RandomWalk.run()\\x1b[39m (pid=97669, ip=127.0.0.1,\
\ actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk\
\ object at 0x11bca6290>)\\n File \"/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py\"\
, line 456, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\\
n File \"/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py\"\
, line 401, in __get_result\\n raise self._exception\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py\"\
, line 744, in run\\n await self._sampleEntityAndAddMeasurementsToQueue(\\\
n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py\"\
, line 923, in _sampleEntityAndAddMeasurementsToQueue\\n entities = await anext(iterator)\\\
n ^^^^^^^^^^^^^^^^^^^^^\\n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py\"\
, line 453, in async_wrapper\\n for entity_batch in self._core_iterator_logic(\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py\"\
, line 178, in _core_iterator_logic\\n yielded_rows += one_additional_row\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 81, in __iadd__\\n self.append(row)\\n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 65, in append\\n ser = self._normalize_row(row)\\n ^^^^^^^^^^^^^^^^^^^^^^^^\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 39, in _normalize_row\\n raise ValueError(\"pd.DataFrame input must\
\ contain exactly one row\")\\nValueError: pd.DataFrame input must contain exactly\
\ one row.', recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 594377, tzinfo=datetime.timezone.utc),\
\ event=<OperationResourceEventEnum.FINISHED: 'finished'>, exit_state=<OperationExitStateEnum.ERROR:\
\ 'error'>), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 51, 49, 601615, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED:\
\ 'updated'>)], metadata={'entities_submitted': 1, 'experiments_requested': 1},\
\ operationType=<DiscoveryOperationEnum.SEARCH: 'search'>, operatorIdentifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty')).\n\
\nray.exceptions.RayTaskError(ValueError): \e[36mray::RandomWalk.run()\e[39m (pid=97669,\
\ ip=127.0.0.1, actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk\
\ object at 0x11bca6290>)\n File \"/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py\"\
, line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n\
\ File \"/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py\"\
, line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n\
\ File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py\"\
, line 744, in run\n await self._sampleEntityAndAddMeasurementsToQueue(\n \
\ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py\"\
, line 923, in _sampleEntityAndAddMeasurementsToQueue\n entities = await anext(iterator)\n\
\ ^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py\"\
, line 453, in async_wrapper\n for entity_batch in self._core_iterator_logic(\n\
\ File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py\"\
, line 178, in _core_iterator_logic\n yielded_rows += one_additional_row\n\
\ File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 81, in __iadd__\n self.append(row)\n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 65, in append\n ser = self._normalize_row(row)\n ^^^^^^^^^^^^^^^^^^^^^^^^\n\
\ File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 39, in _normalize_row\n raise ValueError(\"pd.DataFrame input must contain\
\ exactly one row\")\nValueError: pd.DataFrame input must contain exactly one\
\ row\n\nThe above exception was the direct cause of the following exception:\n\
\nTraceback (most recent call last):\n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_orchestrate_core.py\"\
, line 113, in _run_operation_harness\n operation_output: OperationOutput |\
\ None = run_closure()\n ^^^^^^^^^^^^^\n\
\ File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_general_orchestration.py\"\
, line 43, in _run_general_operation_core\n return operation_function(\n \
\ ^^^^^^^^^^^^^^^^^^^\n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/operator.py\"\
, line 185, in trim\n op_output_iterative_modeling = random_walk(\n \
\ ^^^^^^^^^^^^\n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py\"\
, line 1009, in random_walk\n return orchestrate_explore_operation(\n \
\ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_explore_orchestration.py\"\
, line 331, in orchestrate_explore_operation\n operation_output = _run_operation_harness(\n\
\ ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_orchestrate_core.py\"\
, line 158, in _run_operation_harness\n raise OperationException(\norchestrator.core.operation.operation.OperationException:\
\ ('Error raised while executing operation randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399',\
\ OperationResource(version='v1', kind=<CoreResourceKinds.OPERATION: 'operation'>,\
\ created=datetime.datetime(2026, 2, 20, 9, 50, 13, 848179, tzinfo=datetime.timezone.utc),\
\ config=DiscoveryOperationResourceConfiguration(operation=DiscoveryOperationConfiguration(module=OperatorModuleConf(moduleType=<ModuleTypeEnum.OPERATION:\
\ 'operation'>, moduleName='orchestrator.modules.operators.randomwalk', modulePath='.',\
\ moduleClass='RandomWalk', moduleFunction=None), parameters={'samplerConfig':\
\ {'module': {'moduleType': <ModuleTypeEnum.SAMPLER: 'sampler'>, 'moduleName':\
\ 'trim.trim_sampler', 'modulePath': '.', 'moduleClass': 'TrimSampleSelector',\
\ 'moduleFunction': None}, 'parameters': {'autoGluonArgs': {'tabularPredictorArgs':\
\ {'problem_type': 'regression', 'path': 'trim_models'}, 'fitArgs': {'time_limit':\
\ 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}}, 'finalModelAutoGluonArgs':\
\ {'tabularPredictorArgs': {'problem_type': 'regression', 'path': 'trim_models'},\
\ 'fitArgs': {'time_limit': 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}},\
\ 'targetOutput': 'dataset_tokens_per_second_per_gpu', 'outputDirectory': 'trim_models',\
\ 'debugDirectory': 'debug_output', 'iterationSize': 5, 'holdoutSize': 5, 'samplingBudget':\
\ {'minPoints': 6, 'maxPoints': 12}, 'stoppingCriterion': {'enabled': True, 'meanThreshold':\
\ 0.9, 'stdThreshold': 0.75}, 'noPriorParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu',\
\ 'samples': 6, 'batchSize': 1, 'sampling_strategy': 'clhs'}, 'batchSize': 1,\
\ 'noPriorsParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu',\
\ 'samples': 5}, 'taskId': None, 'taskValue': None}}, 'numberEntities': 12, 'batchSize':\
\ 1, 'singleMeasurement': True, 'maxRetries': 0, 'filter': {'filterMode': <FilterModeEnum.noFilter:\
\ 'noFilter'>}}), metadata=ConfigurationMetadata(name=None, description=None,\
\ labels=None, completed operation='Iterative Modeling Operation'), actuatorConfigurationIdentifiers=['actuatorconfiguration-SFTTrainer-78d001a9'],\
\ spaces=['space-dafd6d-941b7b']), identifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399',\
\ status=[OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 848182, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.CREATED:\
\ 'created'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 848482, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.ADDED:\
\ 'added'>), OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 850608, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.STARTED:\
\ 'started'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 50, 13, 850613, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED:\
\ 'updated'>), OperationResourceStatus(message='Operation exited due to the following\
\ error from a Ray Task: \\x1b[36mray::RandomWalk.run()\\x1b[39m (pid=97669, ip=127.0.0.1,\
\ actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk\
\ object at 0x11bca6290>)\\n File \"/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py\"\
, line 456, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\\
n File \"/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py\"\
, line 401, in __get_result\\n raise self._exception\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py\"\
, line 744, in run\\n await self._sampleEntityAndAddMeasurementsToQueue(\\\
n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File \"/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py\"\
, line 923, in _sampleEntityAndAddMeasurementsToQueue\\n entities = await anext(iterator)\\\
n ^^^^^^^^^^^^^^^^^^^^^\\n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py\"\
, line 453, in async_wrapper\\n for entity_batch in self._core_iterator_logic(\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py\"\
, line 178, in _core_iterator_logic\\n yielded_rows += one_additional_row\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 81, in __iadd__\\n self.append(row)\\n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 65, in append\\n ser = self._normalize_row(row)\\n ^^^^^^^^^^^^^^^^^^^^^^^^\\\
n File \"/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py\"\
, line 39, in _normalize_row\\n raise ValueError(\"pd.DataFrame input must\
\ contain exactly one row\")\\nValueError: pd.DataFrame input must contain exactly\
\ one row.', recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 594377, tzinfo=datetime.timezone.utc),\
\ event=<OperationResourceEventEnum.FINISHED: 'finished'>, exit_state=<OperationExitStateEnum.ERROR:\
\ 'error'>), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026,\
\ 2, 20, 9, 51, 49, 601615, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED:\
\ 'updated'>)], metadata={'entities_submitted': 1, 'experiments_requested': 1},\
\ operationType=<DiscoveryOperationEnum.SEARCH: 'search'>, operatorIdentifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty'))\n"
recorded_at: '2026-02-20T09:51:59.718162Z'
- event: updated
recorded_at: '2026-02-20T09:51:59.718286Z'
version: v1
2026-02-20 09:51:59,731 CRITICAL MainThread orch : orchestrate : Error, ('Error raised while executing operation operation-trim-v0.1-9e3ab70d', OperationResource(version='v1', kind=<CoreResourceKinds.OPERATION: 'operation'>, created=datetime.datetime(2026, 2, 20, 9, 50, 9, 481462, tzinfo=datetime.timezone.utc), config=DiscoveryOperationResourceConfiguration(operation=DiscoveryOperationConfiguration(module=OperatorFunctionConf(operationType=<DiscoveryOperationEnum.CHARACTERIZE: 'characterize'>, operatorName='trim'), parameters={'batchSize': 1, 'autoGluonArgs': {'fitArgs': {'time_limit': 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}, 'tabularPredictorArgs': {'problem_type': 'regression'}}, 'iterationSize': 5, 'outputDirectory': 'trim_models', 'stoppingCriterion': {'enabled': True, 'meanThreshold': 0.9, 'stdThreshold': 0.75}, 'noPriorsParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu', 'samples': 5}, 'samplingBudget': {'maxPoints': 12, 'minPoints': 6}, 'targetOutput': 'dataset_tokens_per_second_per_gpu', 'taskId': None, 'taskValue': None}), metadata=ConfigurationMetadata(name=None, description=None, labels=None), actuatorConfigurationIdentifiers=['actuatorconfiguration-SFTTrainer-78d001a9'], spaces=['space-dafd6d-941b7b']), identifier='operation-trim-v0.1-9e3ab70d', status=[OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 9, 481465, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.CREATED: 'created'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 9, 481790, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.ADDED: 'added'>), OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 9, 485564, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.STARTED: 'started'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 9, 485573, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: 'updated'>), OperationResourceStatus(message='Operation exited due to the following error: (\'Error raised while executing operation randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399\', OperationResource(version=\'v1\', kind=<CoreResourceKinds.OPERATION: \'operation\'>, created=datetime.datetime(2026, 2, 20, 9, 50, 13, 848179, tzinfo=datetime.timezone.utc), config=DiscoveryOperationResourceConfiguration(operation=DiscoveryOperationConfiguration(module=OperatorModuleConf(moduleType=<ModuleTypeEnum.OPERATION: \'operation\'>, moduleName=\'orchestrator.modules.operators.randomwalk\', modulePath=\'.\', moduleClass=\'RandomWalk\', moduleFunction=None), parameters={\'samplerConfig\': {\'module\': {\'moduleType\': <ModuleTypeEnum.SAMPLER: \'sampler\'>, \'moduleName\': \'trim.trim_sampler\', \'modulePath\': \'.\', \'moduleClass\': \'TrimSampleSelector\', \'moduleFunction\': None}, \'parameters\': {\'autoGluonArgs\': {\'tabularPredictorArgs\': {\'problem_type\': \'regression\', \'path\': \'trim_models\'}, \'fitArgs\': {\'time_limit\': 10, \'presets\': \'medium\', \'excluded_model_types\': [\'GBM\']}}, \'finalModelAutoGluonArgs\': {\'tabularPredictorArgs\': {\'problem_type\': \'regression\', \'path\': \'trim_models\'}, \'fitArgs\': {\'time_limit\': 10, \'presets\': \'medium\', \'excluded_model_types\': [\'GBM\']}}, \'targetOutput\': \'dataset_tokens_per_second_per_gpu\', \'outputDirectory\': \'trim_models\', \'debugDirectory\': \'debug_output\', \'iterationSize\': 5, \'holdoutSize\': 5, \'samplingBudget\': {\'minPoints\': 6, \'maxPoints\': 12}, \'stoppingCriterion\': {\'enabled\': True, \'meanThreshold\': 0.9, \'stdThreshold\': 0.75}, \'noPriorParameters\': {\'targetOutput\': \'dataset_tokens_per_second_per_gpu\', \'samples\': 6, \'batchSize\': 1, \'sampling_strategy\': \'clhs\'}, \'batchSize\': 1, \'noPriorsParameters\': {\'targetOutput\': \'dataset_tokens_per_second_per_gpu\', \'samples\': 5}, \'taskId\': None, \'taskValue\': None}}, \'numberEntities\': 12, \'batchSize\': 1, \'singleMeasurement\': True, \'maxRetries\': 0, \'filter\': {\'filterMode\': <FilterModeEnum.noFilter: \'noFilter\'>}}), metadata=ConfigurationMetadata(name=None, description=None, labels=None, completed operation=\'Iterative Modeling Operation\'), actuatorConfigurationIdentifiers=[\'actuatorconfiguration-SFTTrainer-78d001a9\'], spaces=[\'space-dafd6d-941b7b\']), identifier=\'randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399\', status=[OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848182, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.CREATED: \'created\'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848482, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.ADDED: \'added\'>), OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850608, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.STARTED: \'started\'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850613, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: \'updated\'>), OperationResourceStatus(message=\'Operation exited due to the following error from a Ray Task: \\x1b[36mray::RandomWalk.run()\\x1b[39m (pid=97669, ip=127.0.0.1, actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk object at 0x11bca6290>)\\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 456, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 744, in run\\n await self._sampleEntityAndAddMeasurementsToQueue(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 923, in _sampleEntityAndAddMeasurementsToQueue\\n entities = await anext(iterator)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 453, in async_wrapper\\n for entity_batch in self._core_iterator_logic(\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 178, in _core_iterator_logic\\n yielded_rows += one_additional_row\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 81, in __iadd__\\n self.append(row)\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 65, in append\\n ser = self._normalize_row(row)\\n ^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 39, in _normalize_row\\n raise ValueError("pd.DataFrame input must contain exactly one row")\\nValueError: pd.DataFrame input must contain exactly one row.\', recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 594377, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.FINISHED: \'finished\'>, exit_state=<OperationExitStateEnum.ERROR: \'error\'>), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 601615, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: \'updated\'>)], metadata={\'entities_submitted\': 1, \'experiments_requested\': 1}, operationType=<DiscoveryOperationEnum.SEARCH: \'search\'>, operatorIdentifier=\'randomwalk-1.5.1.dev27+5d1c8c2.dirty\')).\n\nray.exceptions.RayTaskError(ValueError): \x1b[36mray::RandomWalk.run()\x1b[39m (pid=97669, ip=127.0.0.1, actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk object at 0x11bca6290>)\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 744, in run\n await self._sampleEntityAndAddMeasurementsToQueue(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 923, in _sampleEntityAndAddMeasurementsToQueue\n entities = await anext(iterator)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 453, in async_wrapper\n for entity_batch in self._core_iterator_logic(\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 178, in _core_iterator_logic\n yielded_rows += one_additional_row\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 81, in __iadd__\n self.append(row)\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 65, in append\n ser = self._normalize_row(row)\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 39, in _normalize_row\n raise ValueError("pd.DataFrame input must contain exactly one row")\nValueError: pd.DataFrame input must contain exactly one row\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_orchestrate_core.py", line 113, in _run_operation_harness\n operation_output: OperationOutput | None = run_closure()\n ^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_general_orchestration.py", line 43, in _run_general_operation_core\n return operation_function(\n ^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/operator.py", line 185, in trim\n op_output_iterative_modeling = random_walk(\n ^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 1009, in random_walk\n return orchestrate_explore_operation(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_explore_orchestration.py", line 331, in orchestrate_explore_operation\n operation_output = _run_operation_harness(\n ^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_orchestrate_core.py", line 158, in _run_operation_harness\n raise OperationException(\norchestrator.core.operation.operation.OperationException: (\'Error raised while executing operation randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399\', OperationResource(version=\'v1\', kind=<CoreResourceKinds.OPERATION: \'operation\'>, created=datetime.datetime(2026, 2, 20, 9, 50, 13, 848179, tzinfo=datetime.timezone.utc), config=DiscoveryOperationResourceConfiguration(operation=DiscoveryOperationConfiguration(module=OperatorModuleConf(moduleType=<ModuleTypeEnum.OPERATION: \'operation\'>, moduleName=\'orchestrator.modules.operators.randomwalk\', modulePath=\'.\', moduleClass=\'RandomWalk\', moduleFunction=None), parameters={\'samplerConfig\': {\'module\': {\'moduleType\': <ModuleTypeEnum.SAMPLER: \'sampler\'>, \'moduleName\': \'trim.trim_sampler\', \'modulePath\': \'.\', \'moduleClass\': \'TrimSampleSelector\', \'moduleFunction\': None}, \'parameters\': {\'autoGluonArgs\': {\'tabularPredictorArgs\': {\'problem_type\': \'regression\', \'path\': \'trim_models\'}, \'fitArgs\': {\'time_limit\': 10, \'presets\': \'medium\', \'excluded_model_types\': [\'GBM\']}}, \'finalModelAutoGluonArgs\': {\'tabularPredictorArgs\': {\'problem_type\': \'regression\', \'path\': \'trim_models\'}, \'fitArgs\': {\'time_limit\': 10, \'presets\': \'medium\', \'excluded_model_types\': [\'GBM\']}}, \'targetOutput\': \'dataset_tokens_per_second_per_gpu\', \'outputDirectory\': \'trim_models\', \'debugDirectory\': \'debug_output\', \'iterationSize\': 5, \'holdoutSize\': 5, \'samplingBudget\': {\'minPoints\': 6, \'maxPoints\': 12}, \'stoppingCriterion\': {\'enabled\': True, \'meanThreshold\': 0.9, \'stdThreshold\': 0.75}, \'noPriorParameters\': {\'targetOutput\': \'dataset_tokens_per_second_per_gpu\', \'samples\': 6, \'batchSize\': 1, \'sampling_strategy\': \'clhs\'}, \'batchSize\': 1, \'noPriorsParameters\': {\'targetOutput\': \'dataset_tokens_per_second_per_gpu\', \'samples\': 5}, \'taskId\': None, \'taskValue\': None}}, \'numberEntities\': 12, \'batchSize\': 1, \'singleMeasurement\': True, \'maxRetries\': 0, \'filter\': {\'filterMode\': <FilterModeEnum.noFilter: \'noFilter\'>}}), metadata=ConfigurationMetadata(name=None, description=None, labels=None, completed operation=\'Iterative Modeling Operation\'), actuatorConfigurationIdentifiers=[\'actuatorconfiguration-SFTTrainer-78d001a9\'], spaces=[\'space-dafd6d-941b7b\']), identifier=\'randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399\', status=[OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848182, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.CREATED: \'created\'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848482, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.ADDED: \'added\'>), OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850608, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.STARTED: \'started\'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850613, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: \'updated\'>), OperationResourceStatus(message=\'Operation exited due to the following error from a Ray Task: \\x1b[36mray::RandomWalk.run()\\x1b[39m (pid=97669, ip=127.0.0.1, actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk object at 0x11bca6290>)\\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 456, in result\\n return self.__get_result()\\n ^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result\\n raise self._exception\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 744, in run\\n await self._sampleEntityAndAddMeasurementsToQueue(\\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 923, in _sampleEntityAndAddMeasurementsToQueue\\n entities = await anext(iterator)\\n ^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 453, in async_wrapper\\n for entity_batch in self._core_iterator_logic(\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 178, in _core_iterator_logic\\n yielded_rows += one_additional_row\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 81, in __iadd__\\n self.append(row)\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 65, in append\\n ser = self._normalize_row(row)\\n ^^^^^^^^^^^^^^^^^^^^^^^^\\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 39, in _normalize_row\\n raise ValueError("pd.DataFrame input must contain exactly one row")\\nValueError: pd.DataFrame input must contain exactly one row.\', recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 594377, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.FINISHED: \'finished\'>, exit_state=<OperationExitStateEnum.ERROR: \'error\'>), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 601615, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: \'updated\'>)], metadata={\'entities_submitted\': 1, \'experiments_requested\': 1}, operationType=<DiscoveryOperationEnum.SEARCH: \'search\'>, operatorIdentifier=\'randomwalk-1.5.1.dev27+5d1c8c2.dirty\'))\n', recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 59, 718162, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.FINISHED: 'finished'>, exit_state=<OperationExitStateEnum.ERROR: 'error'>), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 59, 718286, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: 'updated'>)], metadata={}, operationType=<DiscoveryOperationEnum.CHARACTERIZE: 'characterize'>, operatorIdentifier='trim-v0.1')), detected during operation
⠸ Shutdown - shutting down Ray2026-02-20 09:52:00,889 INFO MainThread orch : graceful_orchestrate_shutdown: Waiting for logs to flush ...
⠧ Shutdown - waiting for logs to flush2026-02-20 09:52:10,894 INFO MainThread orch : graceful_orchestrate_shutdown: Graceful shutdown complete
ERROR: An unexpected error occurred. Operation operation-trim-v0.1-9e3ab70d did not run successfully:
Operation exited due to the following error: ('Error raised while executing operation randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399', OperationResource(version='v1', kind=<CoreResourceKinds.OPERATION: 'operation'>, created=datetime.datetime(2026, 2, 20, 9, 50, 13, 848179, tzinfo=datetime.timezone.utc), config=DiscoveryOperationResourceConfiguration(operation=DiscoveryOperationConfiguration(module=OperatorModuleConf(moduleType=<ModuleTypeEnum.OPERATION: 'operation'>, moduleName='orchestrator.modules.operators.randomwalk', modulePath='.', moduleClass='RandomWalk', moduleFunction=None), parameters={'samplerConfig': {'module': {'moduleType': <ModuleTypeEnum.SAMPLER: 'sampler'>, 'moduleName': 'trim.trim_sampler', 'modulePath': '.', 'moduleClass': 'TrimSampleSelector', 'moduleFunction': None}, 'parameters': {'autoGluonArgs': {'tabularPredictorArgs': {'problem_type': 'regression', 'path': 'trim_models'}, 'fitArgs': {'time_limit': 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}}, 'finalModelAutoGluonArgs': {'tabularPredictorArgs': {'problem_type': 'regression', 'path': 'trim_models'}, 'fitArgs': {'time_limit': 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}}, 'targetOutput': 'dataset_tokens_per_second_per_gpu', 'outputDirectory': 'trim_models', 'debugDirectory': 'debug_output', 'iterationSize': 5, 'holdoutSize': 5, 'samplingBudget': {'minPoints': 6, 'maxPoints': 12}, 'stoppingCriterion': {'enabled': True, 'meanThreshold': 0.9, 'stdThreshold': 0.75}, 'noPriorParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu', 'samples': 6, 'batchSize': 1, 'sampling_strategy': 'clhs'}, 'batchSize': 1, 'noPriorsParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu', 'samples': 5}, 'taskId': None, 'taskValue': None}}, 'numberEntities': 12, 'batchSize': 1, 'singleMeasurement': True, 'maxRetries': 0, 'filter': {'filterMode': <FilterModeEnum.noFilter: 'noFilter'>}}), metadata=ConfigurationMetadata(name=None, description=None, labels=None, completed operation='Iterative Modeling Operation'), actuatorConfigurationIdentifiers=['actuatorconfiguration-SFTTrainer-78d001a9'], spaces=['space-dafd6d-941b7b']), identifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399', status=[OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848182, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.CREATED: 'created'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848482, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.ADDED: 'added'>), OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850608, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.STARTED: 'started'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850613, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: 'updated'>), OperationResourceStatus(message='Operation exited due to the following error from a Ray Task: \x1b[36mray::RandomWalk.run()\x1b[39m (pid=97669, ip=127.0.0.1, actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk object at 0x11bca6290>)\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 744, in run\n await self._sampleEntityAndAddMeasurementsToQueue(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 923, in _sampleEntityAndAddMeasurementsToQueue\n entities = await anext(iterator)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 453, in async_wrapper\n for entity_batch in self._core_iterator_logic(\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 178, in _core_iterator_logic\n yielded_rows += one_additional_row\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 81, in __iadd__\n self.append(row)\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 65, in append\n ser = self._normalize_row(row)\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 39, in _normalize_row\n raise ValueError("pd.DataFrame input must contain exactly one row")\nValueError: pd.DataFrame input must contain exactly one row.', recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 594377, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.FINISHED: 'finished'>, exit_state=<OperationExitStateEnum.ERROR: 'error'>), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 601615, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: 'updated'>)], metadata={'entities_submitted': 1, 'experiments_requested': 1}, operationType=<DiscoveryOperationEnum.SEARCH: 'search'>, operatorIdentifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty')).
ray.exceptions.RayTaskError(ValueError): [36mray::RandomWalk.run()[39m (pid=97669, ip=127.0.0.1, actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk object at 0x11bca6290>)
File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 456, in result
return self.__get_result()
^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
raise self._exception
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 744, in run
await self._sampleEntityAndAddMeasurementsToQueue(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 923, in _sampleEntityAndAddMeasurementsToQueue
entities = await anext(iterator)
^^^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 453, in async_wrapper
for entity_batch in self._core_iterator_logic(
File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 178, in _core_iterator_logic
yielded_rows += one_additional_row
File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 81, in __iadd__
self.append(row)
File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 65, in append
ser = self._normalize_row(row)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 39, in _normalize_row
raise ValueError("pd.DataFrame input must contain exactly one row")
ValueError: pd.DataFrame input must contain exactly one row
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_orchestrate_core.py", line 113, in _run_operation_harness
operation_output: OperationOutput | None = run_closure()
^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_general_orchestration.py", line 43, in _run_general_operation_core
return operation_function(
^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/operator.py", line 185, in trim
op_output_iterative_modeling = random_walk(
^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 1009, in random_walk
return orchestrate_explore_operation(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_explore_orchestration.py", line 331, in orchestrate_explore_operation
operation_output = _run_operation_harness(
^^^^^^^^^^^^^^^^^^^^^^^
File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/_orchestrate_core.py", line 158, in _run_operation_harness
raise OperationException(
orchestrator.core.operation.operation.OperationException: ('Error raised while executing operation randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399', OperationResource(version='v1', kind=<CoreResourceKinds.OPERATION: 'operation'>, created=datetime.datetime(2026, 2, 20, 9, 50, 13, 848179, tzinfo=datetime.timezone.utc), config=DiscoveryOperationResourceConfiguration(operation=DiscoveryOperationConfiguration(module=OperatorModuleConf(moduleType=<ModuleTypeEnum.OPERATION: 'operation'>, moduleName='orchestrator.modules.operators.randomwalk', modulePath='.', moduleClass='RandomWalk', moduleFunction=None), parameters={'samplerConfig': {'module': {'moduleType': <ModuleTypeEnum.SAMPLER: 'sampler'>, 'moduleName': 'trim.trim_sampler', 'modulePath': '.', 'moduleClass': 'TrimSampleSelector', 'moduleFunction': None}, 'parameters': {'autoGluonArgs': {'tabularPredictorArgs': {'problem_type': 'regression', 'path': 'trim_models'}, 'fitArgs': {'time_limit': 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}}, 'finalModelAutoGluonArgs': {'tabularPredictorArgs': {'problem_type': 'regression', 'path': 'trim_models'}, 'fitArgs': {'time_limit': 10, 'presets': 'medium', 'excluded_model_types': ['GBM']}}, 'targetOutput': 'dataset_tokens_per_second_per_gpu', 'outputDirectory': 'trim_models', 'debugDirectory': 'debug_output', 'iterationSize': 5, 'holdoutSize': 5, 'samplingBudget': {'minPoints': 6, 'maxPoints': 12}, 'stoppingCriterion': {'enabled': True, 'meanThreshold': 0.9, 'stdThreshold': 0.75}, 'noPriorParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu', 'samples': 6, 'batchSize': 1, 'sampling_strategy': 'clhs'}, 'batchSize': 1, 'noPriorsParameters': {'targetOutput': 'dataset_tokens_per_second_per_gpu', 'samples': 5}, 'taskId': None, 'taskValue': None}}, 'numberEntities': 12, 'batchSize': 1, 'singleMeasurement': True, 'maxRetries': 0, 'filter': {'filterMode': <FilterModeEnum.noFilter: 'noFilter'>}}), metadata=ConfigurationMetadata(name=None, description=None, labels=None, completed operation='Iterative Modeling Operation'), actuatorConfigurationIdentifiers=['actuatorconfiguration-SFTTrainer-78d001a9'], spaces=['space-dafd6d-941b7b']), identifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty-dcc399', status=[OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848182, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.CREATED: 'created'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 848482, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.ADDED: 'added'>), OperationResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850608, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.STARTED: 'started'>, exit_state=None), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 50, 13, 850613, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: 'updated'>), OperationResourceStatus(message='Operation exited due to the following error from a Ray Task: \x1b[36mray::RandomWalk.run()\x1b[39m (pid=97669, ip=127.0.0.1, actor_id=3d0acc98791e22cbcee5845001000000, repr=<orchestrator.modules.operators.randomwalk.RandomWalk object at 0x11bca6290>)\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 456, in result\n return self.__get_result()\n ^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/.local/share/uv/python/cpython-3.11.11-macos-aarch64-none/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result\n raise self._exception\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 744, in run\n await self._sampleEntityAndAddMeasurementsToQueue(\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/orchestrator/modules/operators/randomwalk.py", line 923, in _sampleEntityAndAddMeasurementsToQueue\n entities = await anext(iterator)\n ^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 453, in async_wrapper\n for entity_batch in self._core_iterator_logic(\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/trim_sampler.py", line 178, in _core_iterator_logic\n yielded_rows += one_additional_row\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 81, in __iadd__\n self.append(row)\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 65, in append\n ser = self._normalize_row(row)\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File "/Users/vassiliad/projects/orchestrator/ado/plugins/operators/trim/trim/utils/rowsring.py", line 39, in _normalize_row\n raise ValueError("pd.DataFrame input must contain exactly one row")\nValueError: pd.DataFrame input must contain exactly one row.', recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 594377, tzinfo=datetime.timezone.utc), event=<OperationResourceEventEnum.FINISHED: 'finished'>, exit_state=<OperationExitStateEnum.ERROR: 'error'>), ADOResourceStatus(message=None, recorded_at=datetime.datetime(2026, 2, 20, 9, 51, 49, 601615, tzinfo=datetime.timezone.utc), event=<ADOResourceEventEnum.UPDATED: 'updated'>)], metadata={'entities_submitted': 1, 'experiments_requested': 1}, operationType=<DiscoveryOperationEnum.SEARCH: 'search'>, operatorIdentifier='randomwalk-1.5.1.dev27+5d1c8c2.dirty'))