Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Sim, make $] Easier benchmarking, via new 'multisim' CLI command #779

Closed
4 tasks done
trentmc opened this issue Mar 12, 2024 · 1 comment
Closed
4 tasks done

[Sim, make $] Easier benchmarking, via new 'multisim' CLI command #779

trentmc opened this issue Mar 12, 2024 · 1 comment
Assignees
@trentmc
Labels
Type: Enhancement New feature or request

Comments

@trentmc
Copy link
Member

trentmc commented Mar 12, 2024 •
edited
Loading

Background / motivation

The challenge: we want to know the effects of various ppss parameters in simulation.

We want to run sweeps across >1 ppss params, and output key results ($ made, accuracy, other) into a csv table.

Candidate approaches: (a) user does each run manually, (b) user has their own custom setups, (c) put it inside 'sim' CLI command, and (d) new 'mu' CLI command.

We'll use approach (d). Best UX, and effort to build it isn't high.

Q: why 'multisim' and not eg 'simsweep'?
A: it makes it easier for us to extend the meta-algorithm that's doing sim-in-loop, while retaining the same ppss setup that specifies what parameters to sweep across (and values to test)

  • For this issue, it's simply a grid search that outputs to csv.
  • Soon, we will use an optimizer to auto-discover the best-performing combination of params.
  • It also opens up to having "fast sweep" that uses model-building to adaptively sample

TODOs / DoD

  • First cut: most things end-to-end. Refactoring at cli level.
  • multisim_ss properly iterates through combos, via PointMeta as multibit counter
  • All end-to-end. Includes: Properly updates PPSS each time; this means maybe getting deeper into dot notation.
  • Support different multifeeds
  • Optimize: multiprocess --> moved to #804
@trentmc trentmc added the Type: Enhancement New feature or request label Mar 12, 2024
@trentmc trentmc self-assigned this Mar 12, 2024
@trentmc trentmc changed the title [Sim, make $] Easier benchmarking, via new 'simsweep' CLI command [Sim, make $] Easier benchmarking, via new 'multisim' CLI command Mar 13, 2024
@trentmc trentmc changed the title [Sim, make $] Easier benchmarking, via new 'multisim' CLI command [Sim, make $] Easier benchmarking, via new 'simsweep' CLI command Mar 13, 2024
@trentmc trentmc changed the title [Sim, make $] Easier benchmarking, via new 'simsweep' CLI command [Sim, make $] Easier benchmarking, via new 'multisim' CLI command Mar 13, 2024
trentmc added a commit that referenced this issue Mar 13, 2024
@trentmc
Fix #779: [Sim, make $] Easier benchmarking, via new 'multisim' CLI c…
7084456 
…ommand
This was referenced Mar 13, 2024
Merged
Closed
trentmc added a commit that referenced this issue Mar 15, 2024
@trentmc
Towards #779: [Sim, make $] Easier benchmarking, via new 'multisim' C…
e9795f5 
…LI (#783)

Towards #779: everything but multiprocess
- New `MultisimSS` module, plus test
- New `MultisimEngine` module, plus test
- PPSS can now optionally be instantiated directly with a constructor dict `d`
- New `PointMeta` module, to define a space of >1 `Point` (also new).
- New utils: `obj_in_objlist()`, `keyval()`, `flat_to_nested_args()`, ..
- Support different multifeeds. Commit 40a79c4

Also fixes:
- [#797](#797) "[CLI, UX] CLI help is too long for new users". Commit 5a5f288
- [#798](#798) "[Logging, UX] I can't tell which logger is outputting which text". Commit 23f295c
- [#789](#789) "[Warning, Aimodel] test_sim_engine has UndefinedMetricWarning in classification". Commit 3c5a7dc
- [#677](#677) "[Warning, CI] In test_mathutil, get 'FutureWarning: Downcasting object dtype arrays'". Commit c8a9c9e
@trentmc
Copy link
Member Author

trentmc commented Mar 15, 2024

Did all but multiprocess in PR #783, commit e9795f5

@trentmc trentmc mentioned this issue Mar 18, 2024
Closed
1 task
@trentmc trentmc closed this as completed Mar 18, 2024
trentmc added a commit that referenced this issue Mar 18, 2024
@trentmc
Follow-up #779: proper syntax for benchmark sweeps on predictoor_ss.a…
c626b8c 
…imodel_ss.input_feeds
@trentmc trentmc mentioned this issue Apr 9, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@trentmc trentmc

Labels
Type: Enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@trentmc