Training Errors #41

alitokur · 2020-07-01T10:15:11Z

Sir, I've created ./bounce/bounce0.txt and ./bounce/face.npy under {path_to_rp_dennis_posed_004_OBJ}.
Now, I try to train, but get RunTimeError:


(tokurEnv) hamit@hamit-MS-7B49:~/Softwares/environments/PIFu$ python -m apps.train_shape --dataroot /home/hamit/Softwares/environments/PIFu/tempImages --random_flip --random_scale --random_trans
/home/hamit/Softwares/environments/PIFu/lib/data/TrainDataset.py:102: UserWarning: loadtxt: Empty input file: "/home/hamit/Softwares/environments/PIFu/tempImages/val.txt"
  var_subjects = np.loadtxt(os.path.join(self.root, 'val.txt'), dtype=str)
train data size:  180
test data size:  360
initialize network with normal
Using Network:  hgpifu
Traceback (most recent call last):
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 12928) is killed by signal: Killed. 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 183, in <module>
    train(opt)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 90, in train
    for train_idx, train_data in enumerate(train_data_loader):
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 12928) exited unexpectedly

PyTorch version : 1.4.0

The text was updated successfully, but these errors were encountered:

shunsukesaito · 2020-07-01T10:51:05Z

If you use num_workers=0, you should be able to see where this error occurs.

alitokur · 2020-07-01T12:03:56Z

I set it to 0 but have still a error, Sir.

(tokurEnv) hamit@hamit-MS-7B49:~/Softwares/environments/PIFu$ python -m apps.train_shape --dataroot /home/hamit/Softwares/environments/PIFu/tempImages --random_flip --random_scale --random_trans
/home/hamit/Softwares/environments/PIFu/lib/data/TrainDataset.py:102: UserWarning: loadtxt: Empty input file: "/home/hamit/Softwares/environments/PIFu/tempImages/val.txt"
  var_subjects = np.loadtxt(os.path.join(self.root, 'val.txt'), dtype=str)
train data size:  180
test data size:  360
initialize network with normal
Using Network:  hgpifu
Killed

alitokur · 2020-07-01T12:25:26Z

I've tried again and
error:

python -m apps.train_shape --dataroot /home/hamit/Softwares/environments/PIFu/tempImages --random_flip --random_scale --random_trans --num_threads=0
/home/hamit/Softwares/environments/PIFu/lib/data/TrainDataset.py:102: UserWarning: loadtxt: Empty input file: "/home/hamit/Softwares/environments/PIFu/tempImages/val.txt"
  var_subjects = np.loadtxt(os.path.join(self.root, 'val.txt'), dtype=str)
train data size:  180
test data size:  360
initialize network with normal
Using Network:  hgpifu
Traceback (most recent call last):
  File "/home/hamit/anaconda3/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/home/hamit/anaconda3/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 183, in <module>
    train(opt)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 105, in train
    res, error = netG.forward(image_tensor, sample_tensor, calib_tensor, labels=label_tensor)
  File "/home/hamit/Softwares/environments/PIFu/lib/model/HGPIFuNet.py", line 131, in forward
    self.filter(images)
  File "/home/hamit/Softwares/environments/PIFu/lib/model/HGPIFuNet.py", line 63, in filter
    self.im_feat_list, self.tmpx, self.normx = self.image_filter(images)
  File "/home/hamit/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hamit/Softwares/environments/PIFu/lib/model/HGFilters.py", line 129, in forward
    hg = self._modules['m' + str(i)](previous)
  File "/home/hamit/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/hamit/Softwares/environments/PIFu/lib/model/HGFilters.py", line 56, in forward
    return self._forward(self.depth, x)
  File "/home/hamit/Softwares/environments/PIFu/lib/model/HGFilters.py", line 39, in _forward
    low2 = self._forward(level - 1, low1)
  File "/home/hamit/Softwares/environments/PIFu/lib/model/HGFilters.py", line 50, in _forward
    up2 = F.interpolate(low3, scale_factor=2, mode='bicubic', align_corners=True)
  File "/home/hamit/anaconda3/lib/python3.7/site-packages/torch/nn/functional.py", line 2459, in interpolate
    " (got {})".format(input.dim(), mode))
NotImplementedError: Input Error: Only 3D, 4D and 5D input Tensors supported (got 4D) for the modes: nearest | linear | bilinear | trilinear (got bicubic)

alitokur · 2020-07-01T16:48:38Z

I guess, there is a problem right there:

  # NOTE: for newer PyTorch (1.3~), it seems that training results are degraded due to implementation diff in F.grid_sample
        # if the pretrained model behaves weirdly, switch with the commented line.
        # NOTE: I also found that "bicubic" works better.
        up2 = F.interpolate(low3, scale_factor=2, mode='bicubic', align_corners=True)
        # up2 = F.interpolate(low3, scale_factor=2, mode='nearest)

I tried the other up2 but then I see a type error:
grid_sample() got an unexpected keyword argument 'align_corners'

shunsukesaito · 2020-07-01T21:14:38Z

Are you sure you use PyTorch 1.4? The newer pytorch should have this 'align_corners' argument. Please try with the latest pytorch if you keep getting this error.

alitokur · 2020-07-02T11:40:38Z

i upgraded pytorch, now;

(tokurEnv) hamit@hamit-MS-7B49:~/Softwares/environments/PIFu$ python -c "import torch; print(torch.__version__)"
1.5.1
and
nvidia-smi :

if i set num_threads=0;

(tokurEnv) hamit@hamit-MS-7B49:~/Softwares/environments/PIFu$ python -m apps.train_shape --dataroot /home/hamit/Softwares/environments/PIFu/tempImages --random_flip --random_scale --random_trans --num_threads=0
/home/hamit/Softwares/environments/PIFu/lib/data/TrainDataset.py:102: UserWarning: loadtxt: Empty input file: "/home/hamit/Softwares/environments/PIFu/tempImages/val.txt"
  var_subjects = np.loadtxt(os.path.join(self.root, 'val.txt'), dtype=str)
train data size:  180
test data size:  360
initialize network with normal
Using Network:  hgpifu
Killed

other way just run the following command:

python -m apps.train_shape --dataroot /home/hamit/Softwares/environments/PIFu/tempImages --random_flip --random_scale --random_trans 
/home/hamit/Softwares/environments/PIFu/lib/data/TrainDataset.py:102: UserWarning: loadtxt: Empty input file: "/home/hamit/Softwares/environments/PIFu/tempImages/val.txt"
  var_subjects = np.loadtxt(os.path.join(self.root, 'val.txt'), dtype=str)
train data size:  180
test data size:  360
initialize network with normal
Using Network:  hgpifu
Traceback (most recent call last):
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 761, in _try_get_data
    data = self._data_queue.get(timeout=timeout)
  File "/usr/lib/python3.6/multiprocessing/queues.py", line 104, in get
    if not self._poll(timeout):
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 257, in poll
    return self._poll(timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 414, in _poll
    r = wait([self], timeout)
  File "/usr/lib/python3.6/multiprocessing/connection.py", line 911, in wait
    ready = selector.select(timeout)
  File "/usr/lib/python3.6/selectors.py", line 376, in select
    fd_event_list = self._poll.poll(timeout)
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/_utils/signal_handling.py", line 66, in handler
    _error_if_any_worker_fails()
RuntimeError: DataLoader worker (pid 5784) is killed by signal: Killed. 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 183, in <module>
    train(opt)
  File "/home/hamit/Softwares/environments/PIFu/apps/train_shape.py", line 90, in train
    for train_idx, train_data in enumerate(train_data_loader):
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 345, in __next__
    data = self._next_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 841, in _next_data
    idx, data = self._get_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 808, in _get_data
    success, data = self._try_get_data()
  File "/home/hamit/Softwares/environments/tokurEnv/lib/python3.6/site-packages/torch/utils/data/dataloader.py", line 774, in _try_get_data
    raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 5784) exited unexpectedly

I'm stuck a little bit Sir.

shunsukesaito · 2020-07-02T11:50:06Z

Okay. Can you spot where the program get stuck inside TrainDataset.py? What if you simply call elements from the dataset without wrapping it with Dataloader?

alitokur · 2020-07-06T15:08:21Z

took me all weekend but I got it.

updated Nvidia driver (default Cuda kit 10.1)
install Cuda 10.1
PyTorch 1.5 with Cuda 10.1
And I'm not sure but it's not working with python 3.6 but after installing Conda ( btw it comes with python 3.7 ) train is working.

hani1994a · 2022-07-27T06:56:49Z

I solved this problem with reduce number of samples in option.py file

alitokur closed this as completed Jul 6, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Training Errors #41

Training Errors #41

alitokur commented Jul 1, 2020

shunsukesaito commented Jul 1, 2020

alitokur commented Jul 1, 2020 •

edited

alitokur commented Jul 1, 2020 •

edited

alitokur commented Jul 1, 2020

shunsukesaito commented Jul 1, 2020

alitokur commented Jul 2, 2020

shunsukesaito commented Jul 2, 2020

alitokur commented Jul 6, 2020

hani1994a commented Jul 27, 2022 •

edited

Training Errors #41

Training Errors #41

Comments

alitokur commented Jul 1, 2020

shunsukesaito commented Jul 1, 2020

alitokur commented Jul 1, 2020 • edited

alitokur commented Jul 1, 2020 • edited

alitokur commented Jul 1, 2020

shunsukesaito commented Jul 1, 2020

alitokur commented Jul 2, 2020

shunsukesaito commented Jul 2, 2020

alitokur commented Jul 6, 2020

hani1994a commented Jul 27, 2022 • edited

alitokur commented Jul 1, 2020 •

edited

alitokur commented Jul 1, 2020 •

edited

hani1994a commented Jul 27, 2022 •

edited