Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

backfill process for old debug.log files #10

Open
josibake opened this issue Oct 20, 2022 · 0 comments
Open

backfill process for old debug.log files #10

josibake opened this issue Oct 20, 2022 · 0 comments

Comments

@josibake
Copy link
Contributor

it would be really handy to be able to run old log files through bmon for stuff where historical data is needed (think old mempool logs). at the very least, these can be scripts and not apart of the actual bmon infra, but i think it makes sense to have them in this repo, at the very least. its also a good way to test bmons log parsing over different versions of core.

a few considerations:

  • if we can just load a log file and have bmon parse it, it might be good to separate it into its own queue so it doesnt affect the live data collection
  • it might also make sense to have a bmon-cli which allows us to access log parsers and write our custom logic of where to put the data (either in the postgres database or some offline storage like GCP), but this way we ensure that all of the logic to parse the data is consistent across time
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant