Skip to content

Benchmark launcher finalization rewrites completed instance results #608

@R-Palazzo

Description

@R-Palazzo

Environment Details

  • SDGym: v0.14.3

Error Description

When a benchmark run times out, the benchmark launcher finalizes the run by filling in missing results*.csv files from available per-job artifacts. However, BenchmarkLauncher.finalize() processes every launched instance, including instances that already completed successfully and already uploaded both results*.csv and metainfo*.yaml.

For those completed instances, finalization should leave results*.csv and metainfo*.yaml untouched. The finalized completed_date format is also inconsistent with normal benchmark completion:

  • normal benchmark path: %m_%d_%Y %H:%M:%S
  • launcher finalize path: %d_%m_%Y %H:%M:%S

Expected behavior:

  • Completed instances should be skipped during finalization.
  • Existing results*.csv files should not be rewritten.
  • Existing completed metainfo*.yaml files should not have completed_date changed.
  • Finalization timestamps should use the same format as normal benchmark completion.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions