Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot inspect the model using TensorBoard #47

Closed
lollows opened this issue Apr 9, 2021 · 4 comments
Closed

Cannot inspect the model using TensorBoard #47

lollows opened this issue Apr 9, 2021 · 4 comments

Comments

@lollows
Copy link

lollows commented Apr 9, 2021

I want to visualize the model using tensorboard. And some error happens:

C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\tensor.py:455: RuntimeWarning: Iterating over a tensor might cause the trace to be incorrect. Passing a tensor of different shape won't change the number of iterations executed (and might lead to errors or silently give incorrect results).
  'incorrect results).', category=RuntimeWarning)
Traceback (most recent call last):
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 549, in compile
    nvrtc.compileProgram(self.ptr, options)
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 112, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 124, in cupy_backends.cuda.libs.nvrtc.compileProgram
  File "cupy_backends\cuda\libs\nvrtc.pyx", line 58, in cupy_backends.cuda.libs.nvrtc.check_status
cupy_backends.cuda.libs.nvrtc.NVRTCError: NVRTC_ERROR_COMPILATION (6)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 354, in <module>
    writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\writer.py", line 714, in add_graph
    self._get_file_writer().add_graph(graph(model, input_to_model, verbose))
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\utils\tensorboard\_pytorch_graph.py", line 285, in graph
    trace = torch.jit.trace(model, args)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 955, in trace
    check_tolerance, strict, _force_outplace, _module_class)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\jit\__init__.py", line 1109, in trace_module
    module._c._create_method_from_trace(method_name, func, example_inputs, var_lookup_fn, strict, _force_outplace)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 307, in forward
    objEstimate = self.netSix(tenFirst[-1], tenSecond[-1], None)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 720, in _call_impl
    result = self._slow_forward(*input, **kwargs)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\torch\nn\modules\module.py", line 704, in _slow_forward
    result = self.forward(*input, **kwargs)
  File "D:/workspace/pycharm/pytorch-pwc-master/check_model.py", line 215, in forward
    input=correlation.FunctionCorrelation(tenFirst=tenFirst, tenSecond=tenSecond),
  File "./correlation\correlation.py", line 386, in FunctionCorrelation
    return _FunctionCorrelation.apply(tenFirst, tenSecond)
  File "./correlation\correlation.py", line 295, in forward
    'output': rbot0
  File "cupy\_util.pyx", line 59, in cupy._util.memoize.decorator.ret
  File "./correlation\correlation.py", line 275, in cupy_launch
    return cupy.cuda.compile_with_cache(strKernel).get_function(strFunction)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 373, in compile_with_cache
    cache_in_memory)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 437, in _compile_with_cache_cuda
    log_stream, cache_in_memory)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 212, in compile_using_nvrtc
    name_expressions, log_stream)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 195, in _compile
    ptx, mapping = prog.compile(options, log_stream)
  File "C:\Users\Administrator\anaconda3\envs\pwc\lib\site-packages\cupy\cuda\compiler.py", line 561, in compile
    'nvrtc' if not runtime.is_hip else 'hiprtc')
cupy.cuda.compiler.CompileException: C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu(16): error: identifier "tensor" is undefined

1 error detected in the compilation of "C:\Users\ADMINI~1\AppData\Local\Temp\tmpthz_eabd\256b2afebe918404edfb6592392d441b_2.cubin.cu".

Here is some of the code I used.

from torch.utils.tensorboard import SummaryWriter

writer = SummaryWriter('runs/test')
writer.add_graph(netNetwork, (tenPreprocessedFirst, tenPreprocessedSecond))
writer.close()

I have tested the run.py, it works well. Please let me know if I missed something. Thank you.

@sniklaus
Copy link
Owner

sniklaus commented Apr 9, 2021

I am afraid that my time is limited and I am unable to handle such support requests, my apologies.

@sniklaus sniklaus closed this as completed Apr 9, 2021
@Etienne66 Etienne66 mentioned this issue Mar 7, 2022
sniklaus added a commit that referenced this issue Mar 9, 2022
@sniklaus
Copy link
Owner

sniklaus commented Mar 9, 2022

Seems like thanks to @Etienne66 this might work now?

@Etienne66
Copy link
Contributor

Here is one of the Graphs that I was able to create.

2022-03-09T030431flow_pwc_test_epoch_0

@sniklaus
Copy link
Owner

sniklaus commented Mar 9, 2022

Looks awesome, thanks for sharing!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants