evaluation not releasing memory

[djtrip]

I noticed that the evaluate_or_sample function doesn't release memory between checkpoints.

For example, if I run ddsp_run in eval mode while another ddsp_run is training, the size of the evaluation process keeps growing as it loads and evaluates the checkpoints that are being generated by the training process. While killing and rerunning the eval process solves the issue, it is not an ideal solution.

[jesseengel]

That's an interesting issue, we haven't run into it before. Are you running eval and training on a single machine? Our setups all run on separate instances reading/writing on a shared filespace. It's fairly straightforward to set something like that up on cloud, with two instances using the same bucket.

[jesseengel]

Just closing this because three aren't more details, but feel free to reopen if it's still a problem.