R/W generated dataset to/from disk #738
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
zhixiangteoh
force-pushed
the
721-diskrw-gsimages
branch
2 times, most recently
from
bbfaf2f
to
4933705
Compare
Codecov Report
@@ Coverage Diff @@
## master #738 +/- ##
==========================================
+ Coverage 88.33% 88.41% +0.07%
==========================================
Files 13 14 +1
Lines 1132 1252 +120
==========================================
+ Hits 1000 1107 +107
- Misses 132 145 +13
Flags with carried forward coverage won't be shown. Click here to find out more.
📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
zhixiangteoh
changed the title
6 tasks
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
reviewed
May 5, 2023
jeff-regier
requested changes
May 5, 2023
|
Also, please make sure training iterations / second is at least 5.5. |
jeff-regier
reviewed
May 5, 2023
zhixiangteoh
force-pushed
the
721-diskrw-gsimages
branch
from
8d62140
to
4a2cde0
Compare
Currently, averaging to about 4 training iterations / second (in each epoch, starts out slower, then speeds up to > 5 it/s). |
jeff-regier
reviewed
May 8, 2023
DiskDataset renders images (in disk_batch_size) chunks and writes/reads them to/from disk via pickle. Encoder uses this DiskDataset in ImagePrior.batch_size batches. Currently does not handle multiple workers—potentially want multiple workers to write/read at the same time.
Also fix formatting.
`case_studies/721_decoder_speedup/main.py mode=generate` renders images via decoder and writes to pkl files on disk. Later `.../main.py mode=train` uses cached image data for training.
DiskDataset renders images (in disk_batch_size) chunks and writes/reads them to/from disk via pickle. Encoder uses this DiskDataset in ImagePrior.batch_size batches. Currently does not handle multiple workers—potentially want multiple workers to write/read at the same time.
Also fix formatting.
`case_studies/721_decoder_speedup/main.py mode=generate` renders images via decoder and writes to pkl files on disk. Later `.../main.py mode=train` uses cached image data for training.
Shuffles disk-cached dataset on each epoch via `random.shuffle` when creating iterator in `__iter__`.
zhixiangteoh
force-pushed
the
721-diskrw-gsimages
branch
from
6e33e9d
to
dca91dc
Compare
With configuration as in gives median 6.1 it/s. |
jeff-regier
reviewed
May 10, 2023
jeff-regier
reviewed
May 10, 2023
bliss/simulator/simulated_dataset.py
Outdated
| return DataLoader(self.valid, batch_size=None, num_workers=0) | ||
|
|
||
| def test_dataloader(self): | ||
| return DataLoader(self, batch_size=None, num_workers=0) |
There was a problem hiding this comment.
Is the test set the same as the training set? That seems like a problem. We should probably have separate files that contain the test set.
jeff-regier
reviewed
May 10, 2023
jeff-regier
reviewed
May 10, 2023
Excellent! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Two-step process.
generator.pyinstantiates and uses aSimulatedDatasetobject to render images incfg.simulator.prior.batch_sizeminibatches, then processes these generated minibatches by concatenation and flattening, before writing out to.pklfiles viapickle.training.use_cached_simulator=trueconfig causes the training step to usedatamodule=instantiate(cfg.cached_simulator)in PyTorch Lightning'strainer.fit.Addresses #721.