Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataLoader worker (pid(s) 9504, 11496, 19564, 12924, 14856, 1256, 20244, 7752) exited unexpectedly #8

Open
batraamul opened this issue Sep 16, 2022 · 8 comments

Comments

@batraamul
Copy link

RuntimeError Traceback (most recent call last)
e:\esrt\train.py in
244 epoch_start = datetime.datetime.now()
245 valid(args.scale)
--> 246 train(epoch)
247 if epoch%10==0:
248 save_checkpoint(epoch)

e:\esrt\train.py in train(epoch)
131 utils.adjust_learning_rate(optimizer, epoch, args.step_size, args.lr, args.gamma)
132 print('epoch =', epoch, 'lr = ', optimizer.param_groups[0]['lr'])
--> 133 for iteration, (lr_tensor, hr_tensor) in enumerate(training_data_loader, 1):
134
135 if args.cuda:

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in next(self)
519 if self._sampler_iter is None:
520 self._reset()
--> 521 data = self._next_data()
522 self._num_yielded += 1
523 if self._dataset_kind == _DatasetKind.Iterable and \

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self)
1184
1185 assert not self._shutdown and self._tasks_outstanding > 0
-> 1186 idx, data = self._get_data()
1187 self._tasks_outstanding -= 1
1188 if self._dataset_kind == _DatasetKind.Iterable:

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _get_data(self)
1150 else:
1151 while True:
-> 1152 success, data = self._try_get_data()
1153 if success:
1154 return data

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _try_get_data(self, timeout)
1001 if len(failed_workers) > 0:
1002 pids_str = ', '.join(str(w.pid) for w in failed_workers)
-> 1003 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e
1004 if isinstance(e, queue.Empty):
1005 return (False, None)

RuntimeError: DataLoader worker (pid(s) 9504, 11496, 19564, 12924, 14856, 1256, 20244, 7752) exited unexpectedly

kidly help to resolve this error. I am also not able to use GPU with the code

@WM-XIAO
Copy link

WM-XIAO commented Jul 13, 2023

Did you solve this problem?I'm also having this issue

@batraamul
Copy link
Author

batraamul commented Jul 15, 2023 via email

@IwantNewway
Copy link

RuntimeError Traceback (most recent call last) e:\esrt\train.py in 244 epoch_start = datetime.datetime.now() 245 valid(args.scale) --> 246 train(epoch) 247 if epoch%10==0: 248 save_checkpoint(epoch)

e:\esrt\train.py in train(epoch) 131 utils.adjust_learning_rate(optimizer, epoch, args.step_size, args.lr, args.gamma) 132 print('epoch =', epoch, 'lr = ', optimizer.param_groups[0]['lr']) --> 133 for iteration, (lr_tensor, hr_tensor) in enumerate(training_data_loader, 1): 134 135 if args.cuda:

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in next(self) 519 if self._sampler_iter is None: 520 self._reset() --> 521 data = self._next_data() 522 self._num_yielded += 1 523 if self._dataset_kind == _DatasetKind.Iterable and \

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self) 1184 1185 assert not self._shutdown and self._tasks_outstanding > 0 -> 1186 idx, data = self._get_data() 1187 self._tasks_outstanding -= 1 1188 if self._dataset_kind == _DatasetKind.Iterable:

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _get_data(self) 1150 else: 1151 while True: -> 1152 success, data = self._try_get_data() 1153 if success: 1154 return data

C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _try_get_data(self, timeout) 1001 if len(failed_workers) > 0: 1002 pids_str = ', '.join(str(w.pid) for w in failed_workers) -> 1003 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e 1004 if isinstance(e, queue.Empty): 1005 return (False, None)

RuntimeError: DataLoader worker (pid(s) 9504, 11496, 19564, 12924, 14856, 1256, 20244, 7752) exited unexpectedly

kidly help to resolve this error. I am also not able to use GPU with the code

u can try to set the num_works zero,may be useful

@batraamul
Copy link
Author

batraamul commented Oct 8, 2023 via email

@batraamul
Copy link
Author

set the num_works zero but still facing the same issue

@batraamul
Copy link
Author

RuntimeError Traceback (most recent call last) e:\esrt\train.py in 244 epoch_start = datetime.datetime.now() 245 valid(args.scale) --> 246 train(epoch) 247 if epoch%10==0: 248 save_checkpoint(epoch)
e:\esrt\train.py in train(epoch) 131 utils.adjust_learning_rate(optimizer, epoch, args.step_size, args.lr, args.gamma) 132 print('epoch =', epoch, 'lr = ', optimizer.param_groups[0]['lr']) --> 133 for iteration, (lr_tensor, hr_tensor) in enumerate(training_data_loader, 1): 134 135 if args.cuda:
C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in next(self) 519 if self._sampler_iter is None: 520 self._reset() --> 521 data = self._next_data() 522 self._num_yielded += 1 523 if self._dataset_kind == _DatasetKind.Iterable and
C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _next_data(self) 1184 1185 assert not self._shutdown and self._tasks_outstanding > 0 -> 1186 idx, data = self._get_data() 1187 self._tasks_outstanding -= 1 1188 if self._dataset_kind == _DatasetKind.Iterable:
C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _get_data(self) 1150 else: 1151 while True: -> 1152 success, data = self._try_get_data() 1153 if success: 1154 return data
C:\ProgramData\Anaconda3\envs\ESRT\lib\site-packages\torch\utils\data\dataloader.py in _try_get_data(self, timeout) 1001 if len(failed_workers) > 0: 1002 pids_str = ', '.join(str(w.pid) for w in failed_workers) -> 1003 raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str)) from e 1004 if isinstance(e, queue.Empty): 1005 return (False, None)
RuntimeError: DataLoader worker (pid(s) 9504, 11496, 19564, 12924, 14856, 1256, 20244, 7752) exited unexpectedly
kidly help to resolve this error. I am also not able to use GPU with the code

u can try to set the num_works zero,may be useful

set the num_works zero but still facing the same issue

@IwantNewway
Copy link

set the num_works zero but still facing the same issue

parser.add_argument("--threads", type=int, default=0, #8
help="number of threads for data loading")
set this?

@Cui-ruochu
Copy link

set all code into main.
you know, if name == 'main'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants