You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Hello! I've found a performance issue in /utils: batch() should be called before map(), which could make your program more efficient. Here is the tensorflow document to support it.
Detailed description is listed below:
/pre_process_mnist.py: dataset_train.batch(batch_size)(here) should be called before dataset_train.map(image_rotate_random,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(image_shift_rand,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(image_squish_random,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(image_erase_random,num_parallel_calls=PARALLEL_INPUT_CALLS)(here) and dataset_train.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).
/pre_process_mnist.py: dataset_test.batch(batch_size)(here) should be called before dataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).
/pre_process_smallnorb.py: dataset_train.batch(batch_size)(here) should be called before dataset_train.map(random_patches,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(random_brightness,num_parallel_calls=PARALLEL_INPUT_CALLS)(here), dataset_train.map(random_contrast,num_parallel_calls=PARALLEL_INPUT_CALLS)(here) and dataset_train.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).
/pre_process_smallnorb.py: dataset_test.batch(1)(here) should be called before dataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)(here).
Besides, you need to check the function called in map()(e.g., generator called in dataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)) whether to be affected or not to make the changed code work properly. For example, if generator needs data with shape (x, y, z) as its input before fix, it would require data with shape (batch_size, x, y, z).
Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.
The text was updated successfully, but these errors were encountered:
Thank you for your tip. However, I have a doubt: does inserting batch before all transformation functions reduce the variance of the dataset? I mean, doing as you're saying, all images of a batch go through the same transformation. On the other hand, keeping batch at the end ensures a different random transformation for each image.
Hello! I've found a performance issue in /utils:
batch()
should be called beforemap()
, which could make your program more efficient. Here is the tensorflow document to support it.Detailed description is listed below:
dataset_train.batch(batch_size)
(here) should be called beforedataset_train.map(image_rotate_random,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here),dataset_train.map(image_shift_rand,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here),dataset_train.map(image_squish_random,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here),dataset_train.map(image_erase_random,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here) anddataset_train.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here).dataset_test.batch(batch_size)
(here) should be called beforedataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here).dataset_train.batch(batch_size)
(here) should be called beforedataset_train.map(random_patches,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here),dataset_train.map(random_brightness,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here),dataset_train.map(random_contrast,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here) anddataset_train.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here).dataset_test.batch(1)
(here) should be called beforedataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)
(here).Besides, you need to check the function called in
map()
(e.g.,generator
called indataset_test.map(generator,num_parallel_calls=PARALLEL_INPUT_CALLS)
) whether to be affected or not to make the changed code work properly. For example, ifgenerator
needs data with shape (x, y, z) as its input before fix, it would require data with shape (batch_size, x, y, z).Looking forward to your reply. Btw, I am very glad to create a PR to fix it if you are too busy.
The text was updated successfully, but these errors were encountered: