New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Monitor size of in-memory data (publish, evaluate) #123

Open

bajtos opened this issue Nov 1, 2023 · 0 comments

Assignees

Member

bajtos commented Nov 1, 2023

Start collecting and visualising the following metrics:

spark-publish: What is the size of a measurement batch submitted to web3.storage?
spark-evaluate: What is the size of all measurements we need to process as part of the evaluation of a single round?

bajtos self-assigned this

bajtos changed the title ~~Monitor memory usage (publish, evaluate)~~ Monitor data size (publish, evaluate)

bajtos changed the title ~~Monitor data size (publish, evaluate)~~ Monitor size of in-memory data (publish, evaluate)

bajtos mentioned this issue

Handle large number of measurements filecoin-station/spark-evaluate#5

Open

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment