You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Each game log file costs around 500kb-1mb currently. During long training runs this can accumulate dozens of gb of text files, most of which aren't really that useful to examine unless there's a bug or some extra analysis that's needed after the fact, and for large numbers of threads this may cause a disk I/O bottleneck.
Modify the game simulator threads to keep logs in memory and only write to disk at the end if there's an error, otherwise discard them. This might cause a big memory hit for multithreaded setups but likely still manageable.
Since some bugs are silent, it may also be useful to store a (random?) sample of un-errored logs, e.g. all 2^n-numbered games which should greatly stem disk usage growth for longer training runs.
The text was updated successfully, but these errors were encountered:
Each game log file costs around 500kb-1mb currently. During long training runs this can accumulate dozens of gb of text files, most of which aren't really that useful to examine unless there's a bug or some extra analysis that's needed after the fact, and for large numbers of threads this may cause a disk I/O bottleneck.
Modify the game simulator threads to keep logs in memory and only write to disk at the end if there's an error, otherwise discard them. This might cause a big memory hit for multithreaded setups but likely still manageable.
Since some bugs are silent, it may also be useful to store a (random?) sample of un-errored logs, e.g. all 2^n-numbered games which should greatly stem disk usage growth for longer training runs.
The text was updated successfully, but these errors were encountered: