aspect_ratio_bucket_indices.json file inside dataset folder re-used incorrectly after change in dataset

[zhuliyi0]

Jupyter had that .ipynb_checkpoint folder created inside dataset folder, that crashed training, then I deleted it, but aspect_ratio_bucket_indices.json file still has those files, training crashed again. Went back to normal after deleting json file and let it regenerate.

Better not to re-use that json file? Or have a quick check on the number of files?

[bghira]

we do already run refresh_buckets() at startup, which removes nonexistent files and adds new ones.

is there an issue with how that's functioning?

[zhuliyi0]

Yes the file inside .ipynb_checkpoint that was deleted was not removed inside that json file when a new run started, it showed wrong number of training examples, then errored out at first few step saying some file inside that folder cannot be found. Not sure if this is only caused by .ipynb_checkpoint directory, still getting familiar with Linux.

[bghira]

since i've fixed the ipynb deletion a bit better so it removes from input/output/temp dirs, i'm going to close this out

fixed for v0.4