Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Reduce game log size #351

Closed
taylorhansen opened this issue Jan 7, 2023 · 0 comments
Closed

Reduce game log size #351

taylorhansen opened this issue Jan 7, 2023 · 0 comments
Assignees
Labels
enhancement Something should be changed training Has to do with the training script

Comments

@taylorhansen
Copy link
Owner

taylorhansen commented Jan 7, 2023

Each game log file costs around 500kb-1mb currently. During long training runs this can accumulate dozens of gb of text files, most of which aren't really that useful to examine unless there's a bug or some extra analysis that's needed after the fact, and for large numbers of threads this may cause a disk I/O bottleneck.

Modify the game simulator threads to keep logs in memory and only write to disk at the end if there's an error, otherwise discard them. This might cause a big memory hit for multithreaded setups but likely still manageable.

Since some bugs are silent, it may also be useful to store a (random?) sample of un-errored logs, e.g. all 2^n-numbered games which should greatly stem disk usage growth for longer training runs.

@taylorhansen taylorhansen added enhancement Something should be changed training Has to do with the training script labels Jan 7, 2023
@taylorhansen taylorhansen self-assigned this Jan 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Something should be changed training Has to do with the training script
Projects
None yet
Development

No branches or pull requests

1 participant