Comparing baseline benchmarks #26

mistakia · 2021-01-20T19:05:25Z

To make it easier to compare benchmarks, I was thinking about adding a cli flag that would be used to write the baseline benchmark results to a provided path. Then this file can be loaded to provide a diff between two baseline benchmarks (i.e. comparing a PR to master).

Let me know if there's a better way to do this and how it could possibly integrate with CircleCI so that perhaps we could automate comparing baseline benchmarks for PRs.

aphelionz · 2021-01-20T19:32:09Z

I think this sounds like a good idea as any. Anything that would get us automated comparative benchmarks like this is welcome.

mistakia · 2021-01-20T19:42:17Z

My understanding of CircleCI is very limited but I was imagining it would run the benchmark in master and provide a globally accessible path. Then it would run the benchmark in that branch and the benchmark runner has access to that previous path and can output a report.

Not sure if this is possible.

aphelionz · 2021-01-20T19:55:32Z

Yes, I believe it is via build artifacts. If you find it's easier in a different CI system i.e. GH actions that's fine too. I like CircleCI because I like a healthy ecosystem of tools instead of getting further locked-in to GH, but we're kinda there already anyway :P

mistakia mentioned this issue Jan 20, 2021

Feat: improve memory usage / allow for in-memory storage of all entries to be optional (WIP) orbitdb-archive/ipfs-log#301

Draft

8 tasks

mistakia linked a pull request Feb 2, 2021 that will close this issue

feat: save/compare results to/from file #29

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Comparing baseline benchmarks #26

Comparing baseline benchmarks #26

mistakia commented Jan 20, 2021 •

edited

aphelionz commented Jan 20, 2021

mistakia commented Jan 20, 2021 •

edited

aphelionz commented Jan 20, 2021

Comparing baseline benchmarks #26

Comparing baseline benchmarks #26

Comments

mistakia commented Jan 20, 2021 • edited

aphelionz commented Jan 20, 2021

mistakia commented Jan 20, 2021 • edited

aphelionz commented Jan 20, 2021

mistakia commented Jan 20, 2021 •

edited

mistakia commented Jan 20, 2021 •

edited