import deepwave error #50
Comments
|
Hi Naveen,
Thank you for writing.
The only errors that I see in that output seem to be ones like "Could
not open output file
'/localscratch/1139367/tmpxft_00001ff3_0000000a'". The compiler
appears to want to store temporary files in the /localscratch
directory and my guess is that it is unable to. Is it possible that it
does not have permission to write to that location, or might
/localscratch be full?
|
|
Hi Naveen,
You might like to try setting the TMPDIR environment variable to a
location that nvcc will definitely have write access to (such as a
directory you create in your home directory), as the NVCC compiler
will apparently then use that location for its temporary files. So you
could try "mkdir /home/naveengupta/nvcctmp" and then "export
TMPDIR=/home/naveengupta/nvcctmp; python -c 'import deepwave'"
(assuming you are using bash). Does that work?
|
|
Hi Alan, I tried this command and still getting the same error. Could you please suggest if you have any insights? |
|
Hi, Alan. Traceback (most recent call last): The above exception was the direct cause of the following exception: Traceback (most recent call last): Process finished with exit code 1 |
|
@naveenguptaiitr That is a pity. Can you post the new error message? If it is indeed exactly the same - still trying to write to /localscratch - then it seems that NVCC is not using (or receiving) the updated TMPDIR environment variable. If it is using the new TMPDIR and is still unable to write then there seems to be a different problem. Can you also report what the output is when you run the command "touch $TMPDIR/naveentest" on the system that you are trying to use Deepwave on? |
|
I am able to fix the issue with the import using the commands you shared first. The problem was with the updating the TMPDIR variable. Thank you very much for your suggestion and the import issue is fixed. However, when I am running the FWI inversion on marmousi data, I am getting Location not within the model error. Could you share your insights on this? Thank you again for quickly helping out with the issue. |
|
I am glad to hear that you are now able to use Deepwave.
Are you able to show the code you are using that results in an error?
Location not within the model means that one or more of the source or
receiver locations are outside the velocity model.
|
|
Hi Alan, #################################################### import torch import numpy as np device = torch.device('cuda') Marmousi velocity modelThe velocity model size is (96, 192) -> (nz, ny)vp_true, vp0, m, d = marmousi2(origin=(7500, 355), vp_true = torch.tensor(vp_true) acquisition setupdx = m.deltas[0] num_shots = 20 num_sources_per_shot = 1 num_receivers_per_shot = 96 peak_freq = 15 n_seconds = 4 source_locationsx_s = torch.zeros(num_shots, num_sources_per_shot, num_dims) receivers_locationsx_r = torch.zeros(num_shots, num_receivers_per_shot, num_dims) #source amplitude tensor print(source_amplitudes.shape) predict the amplitude on the marmousi velocity modelreceiver_amplitudes_true = scalar(vp_true, dx, dt, source_amplitudes=source_amplitudes, ################################################################# Error - RuntimeError Traceback (most recent call last) File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar.py:360, in scalar(v, grid_spacing, dt, source_amplitudes, source_locations, receiver_locations, accuracy, pml_width, pml_freq, max_vel, survey_pad, wavefield_0, wavefield_m1, psiy_m1, psix_m1, zetay_m1, zetax_m1, origin, nt, model_gradient_sampling_interval, freq_taper_frac, time_pad_frac) File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:72, in setup_propagator(models, prop_type, grid_spacing, dt, wavefields, source_amplitudes, source_locations, receiver_locations, accuracy, pml_width, pml_freq, max_vel, survey_pad, origin, nt, model_gradient_sampling_interval, freq_taper_frac, time_pad_frac) File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:584, in extract_survey(models, locations, survey_pad, wavefields, origin, pad) File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/common.py:610, in check_locations_are_within_model(model_shape, locations) RuntimeError: Locations must be within model. |
|
Hi Naveen, The problem is that your velocity model seems to be shaped incorrectly. When I run your code, the output of vp_true = vp_true.reshape(96, 192)then it will have the correct shape. Furthermore, Deepwave uses cell indexes as source/receiver locations, so you also need to remove the multiplication by source_spacing = ny / num_shots
receiver_spacing = ny / num_receivers_per_shotIf you need to have locations that are not centered on grid cells then you may wish to look at Deepwave's location interpolation example. After making these changes I was able to run your code, and produced an image of first shot's data using vmin, vmax = np.percentile(receiver_amplitudes_true[-1][0], [5,95])
plt.imshow(receiver_amplitudes_true[-1][0].T, aspect='auto', vmin=vmin, vmax=vmax) |
|
Hi Alan, Thank you very much for your help.! This is working and am delighted to get this working. Appreciate your help. I had few more questions to ask. Could you please address following questions?
Thank you again for helping out with the issue. :) |
|
Hi Naveen, That is good news. The Usage page of the Deepwave documentation describes all of the inputs and outputs. The other outputs relate the the wavefield at the final timestep. One of the reasons that these are provided is so that calls to Deepwave can be chained together, allowing you to continue propagating to later timesteps (as in the checkpointing example). On its own, Deepwave does not provide a way to propagate with different velocity models simultaneously, but you might be able to get that to work by combining it with something else, such as by using Python's or PyTorch's multiprocessing capabilities to run Deepwave in multiple processes. |
|
Hi Alan, Thank you very much for sharing your insights. I will check out the documentation to learn more about the deepwave. It is a great library and indeed super helpful for seismic inversion problems. Thank you for helping out. |
Hi Alan,
I installed the latest version of deepwave and the installation went smoothly. However, I am unable to import the package and getting the error mentioned below. Could you please help? Thank you.!
My torch version is 1.12.0+cu113
import torch
import deepwave
CalledProcessError Traceback (most recent call last)
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1808, in _run_ninja_build(build_directory, verbose, error_prefix)
1807 stdout_fileno = 1
-> 1808 subprocess.run(
1809 command,
1810 stdout=stdout_fileno if verbose else subprocess.PIPE,
1811 stderr=subprocess.STDOUT,
1812 cwd=build_directory,
1813 check=True,
1814 env=env)
1815 except subprocess.CalledProcessError as e:
1816 # Python 2 and 3 compatible way of getting the error object.
File ~/.conda/envs/openfwi/lib/python3.8/subprocess.py:516, in run(input, capture_output, timeout, check, *popenargs, **kwargs)
515 if check and retcode:
--> 516 raise CalledProcessError(retcode, process.args,
517 output=stdout, stderr=stderr)
518 return CompletedProcess(process.args, retcode, stdout, stderr)
CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.
The above exception was the direct cause of the following exception:
RuntimeError Traceback (most recent call last)
Cell In[2], line 1
----> 1 import deepwave
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/init.py:31
27 if torch.cuda.is_available():
28 sources += [source_dir / 'scalar.cu',
29 source_dir / 'scalar_born.cu',
30 source_dir / 'elastic.cu']
---> 31 torch.utils.cpp_extension.load(
32 name="deepwave",
33 sources=sources,
34 is_python_module=False,
35 extra_cflags=['-march=native', '-Ofast', '-fopenmp'],
36 extra_cuda_cflags=['--restrict', '-O3', '--use_fast_math'],
37 extra_ldflags=['-march=native', '-Ofast', '-fopenmp']
38 )
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1202, in load(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
1111 def load(name,
1112 sources: Union[str, List[str]],
1113 extra_cflags=None,
(...)
1121 is_standalone=False,
1122 keep_intermediates=True):
1123 r'''
1124 Loads a PyTorch C++ extension just-in-time (JIT).
1125
(...)
1200 verbose=True)
1201 '''
-> 1202 return _jit_compile(
1203 name,
1204 [sources] if isinstance(sources, str) else sources,
1205 extra_cflags,
1206 extra_cuda_cflags,
1207 extra_ldflags,
1208 extra_include_paths,
1209 build_directory or _get_build_directory(name, verbose),
1210 verbose,
1211 with_cuda,
1212 is_python_module,
1213 is_standalone,
1214 keep_intermediates=keep_intermediates)
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1425, in _jit_compile(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_python_module, is_standalone, keep_intermediates)
1421 hipified_sources.add(hipify_result[s_abs]["hipified_path"] if s_abs in hipify_result else s_abs)
1423 sources = list(hipified_sources)
-> 1425 _write_ninja_file_and_build_library(
1426 name=name,
1427 sources=sources,
1428 extra_cflags=extra_cflags or [],
1429 extra_cuda_cflags=extra_cuda_cflags or [],
1430 extra_ldflags=extra_ldflags or [],
1431 extra_include_paths=extra_include_paths or [],
1432 build_directory=build_directory,
1433 verbose=verbose,
1434 with_cuda=with_cuda,
1435 is_standalone=is_standalone)
1436 finally:
1437 baton.release()
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1537, in _write_ninja_file_and_build_library(name, sources, extra_cflags, extra_cuda_cflags, extra_ldflags, extra_include_paths, build_directory, verbose, with_cuda, is_standalone)
1535 if verbose:
1536 print(f'Building extension module {name}...')
-> 1537 _run_ninja_build(
1538 build_directory,
1539 verbose,
1540 error_prefix=f"Error building extension '{name}'")
File ~/.conda/envs/openfwi/lib/python3.8/site-packages/torch/utils/cpp_extension.py:1824, in _run_ninja_build(build_directory, verbose, error_prefix)
1822 if hasattr(error, 'output') and error.output: # type: ignore[union-attr]
1823 message += f": {error.output.decode(*SUBPROCESS_DECODE_ARGS)}" # type: ignore[union-attr]
-> 1824 raise RuntimeError(message) from e
RuntimeError: Error building extension 'deepwave': [1/4] /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1013" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/elastic.cu -o elastic.cuda.o
FAILED: elastic.cuda.o
/cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1013" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/elastic.cu -o elastic.cuda.o
nvcc fatal : Could not open output file '/localscratch/1139367/tmpxft_00001ff5_0000000a'
[2/4] /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1013" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar_born.cu -o scalar_born.cuda.o
FAILED: scalar_born.cuda.o
/cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1013" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar_born.cu -o scalar_born.cuda.o
nvcc fatal : Could not open output file '/localscratch/1139367/tmpxft_00001ff4_0000000a'
[3/4] /cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1013" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar.cu -o scalar.cuda.o
FAILED: scalar.cuda.o
/cm/shared/apps/cuda11.4/toolkit/11.4.2/bin/nvcc -DTORCH_EXTENSION_NAME=deepwave -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE="gcc" -DPYBIND11_STDLIB="libstdcpp" -DPYBIND11_BUILD_ABI="cxxabi1013" -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/torch/csrc/api/include -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/TH -isystem /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/torch/include/THC -isystem /cm/shared/apps/cuda11.4/toolkit/11.4.2/include -isystem /home/naveengupta/.conda/envs/openfwi/include/python3.8 -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS -D__CUDA_NO_HALF_CONVERSIONS_ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_60,code=sm_60 -gencode=arch=compute_61,code=sm_61 -gencode=arch=compute_70,code=sm_70 -gencode=arch=compute_75,code=sm_75 -gencode=arch=compute_80,code=sm_80 -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 --compiler-options '-fPIC' --restrict -O3 --use_fast_math -std=c++14 -c /home/naveengupta/.conda/envs/openfwi/lib/python3.8/site-packages/deepwave/scalar.cu -o scalar.cuda.o
nvcc fatal : Could not open output file '/localscratch/1139367/tmpxft_00001ff3_0000000a'
ninja: build stopped: subcommand failed.
The text was updated successfully, but these errors were encountered: