-
Notifications
You must be signed in to change notification settings - Fork 613
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix TensorFlow plugin operation without GPU #3719
Conversation
- TensorFlow DALI plugin doesn't work without GPU as it forces synchronization after copying out the output data to the TF tensor what invokes CUDA call. This PR removes this synchronization when we copy to the non-pinned CPU buffer and when the DALI TF operator is placed on the CPU Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
return data | ||
|
||
def get_data(batch_size, value): | ||
pipe = get_datali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pipe = get_datali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) | |
pipe = get_dali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
def test_dali_tf_op_cpu_only(): | ||
try: | ||
tf.compat.v1.disable_eager_execution() | ||
except: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
except: | |
except Exception: |
LGTM was complaining about using "except:" as a bad practice
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
|
||
|
||
@pipeline_def() | ||
def get_datali_pipe(value): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
def get_datali_pipe(value): | |
def get_dali_pipe(value): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
return data | ||
|
||
def get_data(batch_size, value): | ||
pipe = get_datali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pipe = get_datali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) | |
pipe = get_dali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
skip_for_incompatible_tf() | ||
try: | ||
tf.compat.v1.enable_eager_execution() | ||
except: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
except: | |
except Exception: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
|
||
batch_size = 3 | ||
value = random.randint(0, 1000) | ||
pipe = get_datali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
pipe = get_datali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) | |
pipe = get_dali_pipe(batch_size=batch_size, device_id=types.CPU_ONLY_DEVICE_ID, num_threads=1, value=value) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
dali_tf_plugin/dali_dataset_op.cc
Outdated
unsigned int wait_flag = (out_id == num_outputs - 1) ? DALI_ext_force_sync : DALI_ext_default; | ||
// if the OP runs on the CPU the output memory is not pinned and we don't need to sync | ||
unsigned int wait_flag = (i == dali_num_out - 1) ? | ||
(this->device_type_ == device_type_t::CPU ? 0 : DALI_ext_force_sync) : |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
(this->device_type_ == device_type_t::CPU ? 0 : DALI_ext_force_sync) : | |
(this->device_type_ == device_type_t::CPU ? DALI_ext_default : DALI_ext_force_sync) : |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or
unsigned int wait_flag = this->device_type_ != device_type_t::CPU && (i == dali_num_out - 1) ? DALI_ext_force_sync : DALI_ext_default;
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
dali_tf_plugin/daliop.cc
Outdated
// Synchronize with the dataset()->stream_ when doing the last copy, so the outputs | ||
// are fully finished before we release the output buffers for reuse. | ||
unsigned int wait_flag = (i == dali_num_out - 1) ? DALI_ext_force_sync : DALI_ext_default; | ||
// if the OP runs on the CPU the output memory is not pinned and we don't need to sync | ||
unsigned int wait_flag = (i == dali_num_out - 1) ? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
same here
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
qa/TL0_cpu_only/test_pytorch.sh
Outdated
# CPU only test, remove CUDA from the search path just in case | ||
export LD_LIBRARY_PATH="" | ||
export PATH=${PATH/cuda/} | ||
nosetests --verbose test_dali_cpu_only.py |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nosetests --verbose test_dali_cpu_only.py | |
nosetests --verbose -m '(?:^|[\b_\./-])[Tt]est.*pytorch*' test_dali_cpu_only.py |
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Or should we use --attr 'pytorch'
vs. --attr '!pytorch'
? Right now we are running all tests under test_pytorch.sh
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
CI MESSAGE: [4093192]: BUILD FAILED |
CI MESSAGE: [4106587]: BUILD STARTED |
CI MESSAGE: [4106908]: BUILD STARTED |
CI MESSAGE: [4107712]: BUILD STARTED |
CI MESSAGE: [4106908]: BUILD FAILED |
CI MESSAGE: [4108281]: BUILD STARTED |
CI MESSAGE: [4108281]: BUILD FAILED |
CI MESSAGE: [4111696]: BUILD STARTED |
CI MESSAGE: [4111696]: BUILD FAILED |
CI MESSAGE: [4116022]: BUILD STARTED |
CI MESSAGE: [4116022]: BUILD FAILED |
CI MESSAGE: [4117029]: BUILD STARTED |
CI MESSAGE: [4117029]: BUILD FAILED |
CI MESSAGE: [4117747]: BUILD STARTED |
CI MESSAGE: [4117747]: BUILD FAILED |
CI MESSAGE: [4118800]: BUILD STARTED |
CI MESSAGE: [4118800]: BUILD PASSED |
- TensorFlow DALI plugin doesn't work without GPU as it forces synchronization after copying out the output data to the TF tensor what invokes CUDA call. This PR removes this synchronization when we copy to the non-pinned CPU buffer and when the DALI TF operator is placed on the CPU Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
- TensorFlow DALI plugin doesn't work without GPU as it forces synchronization after copying out the output data to the TF tensor what invokes CUDA call. This PR removes this synchronization when we copy to the non-pinned CPU buffer and when the DALI TF operator is placed on the CPU Signed-off-by: Janusz Lisiecki <jlisiecki@nvidia.com>
synchronization after copying out the output data to the TF tensor
what invokes CUDA call. This PR removes this synchronization when
we copy to the non-pinned CPU buffer and when the DALI TF operator
is placed on the CPU
Signed-off-by: Janusz Lisiecki jlisiecki@nvidia.com
Category:
Bug fix (non-breaking change which fixes an issue)
Description:
synchronization after copying out the output data to the TF tensor
what invokes CUDA call. This PR removes this synchronization when
we copy to the non-pinned CPU buffer and when the DALI TF operator
is placed on the CPU
Additional information:
Affected modules and functionalities:
Key points relevant for the review:
Checklist
Tests
Documentation
DALI team only
Requirements
REQ IDs: N/A
JIRA TASK: N/A