Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Unable to Train in Single GPU/CPU Debug Mode on Windows #108

Open
nadolsw opened this issue Jan 24, 2022 · 0 comments
Open

Unable to Train in Single GPU/CPU Debug Mode on Windows #108

nadolsw opened this issue Jan 24, 2022 · 0 comments

Comments

@nadolsw
Copy link

nadolsw commented Jan 24, 2022 •
edited

I am unable to execute the training script in single GPU/CPU debug mode on Windows using the following command:

python train.py -c config.json -d 1 -dist false

It looks like it begins to train:

(pick) C:\Users\winado\Desktop\PICK>python train.py -c config.json -d 1 -dist false
CUDA Devices Detected: 2
PyTorch 1.5.1 _CudaDeviceProperties(name='NVIDIA GeForce RTX 3070', major=8, minor=6, total_memory=8192MB, multi_processor_count=46)
[2022-01-24 11:34:27,686 - train - INFO] - One GPU or CPU training mode start...
[2022-01-24 11:34:27,698 - train - INFO] - Dataloader instances created. Train datasets: 32 samples Validation datasets: 32 samples.
[2022-01-24 11:34:28,391 - train - INFO] - Model created, trainable parameters: 68567386.
[2022-01-24 11:34:28,393 - train - INFO] - Optimizer and lr_scheduler created.
[2022-01-24 11:34:28,393 - train - INFO] - Max_epochs: 100 Log_per_step: 10 Validation_per_step: 50.
[2022-01-24 11:34:28,393 - train - INFO] - Training start...
[2022-01-24 11:34:28,394 - trainer - WARNING] - Training is using GPU 0!

But then produces the following error after about 15 minutes of appearing stalled:

Traceback (most recent call last):
File "", line 1, in
Traceback (most recent call last):
File "", line 1, in
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 304, in _optimize_charset
File "", line 1, in
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel) File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
charmap[k] = 1
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main
IndexError: bytearray index out of range File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
prepare(preparation_data)
exitcode = _main(fd, parent_sentinel) exitcode = _main(fd, parent_sentinel) exitcode = _main(fd, parent_sentinel)
During handling of the above exception, another exception occurred:
exitcode = _main(fd, parent_sentinel)exitcode = _main(fd, parent_sentinel)
...

During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "train.py", line 167, in
entry_point(config)
File "train.py", line 131, in entry_point
main(config, local_master, logger if local_master else None)
File "train.py", line 79, in main
trainer.train()
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 135, in train
result_dict = self._train_epoch(epoch)
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 199, in _train_epoch
for step_idx, input_data_item in enumerate(self.data_loader):
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 345, in next
data = self._next_data()
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 798, in _get_data
success, data = self._try_get_data()
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 21916, 28716, 45548, 29736, 15064, 26248, 14120, 20168) exited unexpectedly

For reference, I do have two nvidia and CUDA-enabled GPU devices connected but they both remain at 0% utilization even as the training script above appears to be "running".

image

I also ensured that I appropriately modified the file paths within the config.json file:

"train_dataset": {
"type": "PICKDataset",
"args": {
"files_name":"C:/users/winado/Desktop/PICK/data/data_examples_root/train_samples_list.csv",
"boxes_and_transcripts_folder":"boxes_and_transcripts",
"images_folder":"images",
"entities_folder":"entities",
"iob_tagging_type":"box_and_within_box_level",
"resized_image_size": [480, 960],
"ignore_error": false
}
},
"validation_dataset": {
"type": "PICKDataset",
"args": {
"files_name":"C:/users/winado/Desktop/PICK/data/data_examples_root/train_samples_list.csv",
"boxes_and_transcripts_folder":"boxes_and_transcripts",
"images_folder":"images",
"entities_folder":"entities",
"iob_tagging_type":"box_and_within_box_level",
"resized_image_size": [480, 960],
"ignore_error": false
}
},

Was hoping someone may be able to provide a bit of guidance on next steps?

The full trace back is below:

(pick) C:\Users\winado\Desktop\PICK>python train.py -c config.json -d 1 -dist false
CUDA Devices Detected: 2
PyTorch 1.5.1 _CudaDeviceProperties(name='NVIDIA GeForce RTX 3070', major=8, minor=6, total_memory=8192MB, multi_processor_count=46)
[2022-01-24 11:34:27,686 - train - INFO] - One GPU or CPU training mode start...
[2022-01-24 11:34:27,698 - train - INFO] - Dataloader instances created. Train datasets: 32 samples Validation datasets: 32 samples.
[2022-01-24 11:34:28,391 - train - INFO] - Model created, trainable parameters: 68567386.
[2022-01-24 11:34:28,393 - train - INFO] - Optimizer and lr_scheduler created.
[2022-01-24 11:34:28,393 - train - INFO] - Max_epochs: 100 Log_per_step: 10 Validation_per_step: 50.
[2022-01-24 11:34:28,393 - train - INFO] - Training start...
[2022-01-24 11:34:28,394 - trainer - WARNING] - Training is using GPU 0!
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "", line 1, in
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
File "", line 1, in
Traceback (most recent call last):
File "", line 1, in
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 304, in _optimize_charset
File "", line 1, in
File "", line 1, in
File "", line 1, in
File "", line 1, in
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel) File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main

charmap[k] = 1
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main
IndexError: bytearray index out of range File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
prepare(preparation_data)
exitcode = _main(fd, parent_sentinel) exitcode = _main(fd, parent_sentinel) exitcode = _main(fd, parent_sentinel)

During handling of the above exception, another exception occurred:

exitcode = _main(fd, parent_sentinel)exitcode = _main(fd, parent_sentinel)
exitcode = _main(fd, parent_sentinel)
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
Traceback (most recent call last):
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main

File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main

File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main
File "", line 1, in
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main
prepare(preparation_data)
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 116, in spawn_main
prepare(preparation_data)_fixup_main_from_path(data['init_main_from_path']) prepare(preparation_data)
exitcode = _main(fd, parent_sentinel)

prepare(preparation_data)

_fixup_main_from_path(data['init_main_from_path'])

File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
prepare(preparation_data) File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 125, in _main

File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
prepare(preparation_data)
prepare(preparation_data)
_fixup_main_from_path(data['init_main_from_path']) File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
main_content = runpy.run_path(main_path,
_fixup_main_from_path(data['init_main_from_path'])main_content = runpy.run_path(main_path,
_fixup_main_from_path(data['init_main_from_path']) File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 236, in prepare

File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path

  File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path

      File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path

File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
_fixup_main_from_path(data['init_main_from_path']) File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
return _run_module_code(code, init_globals, run_name,_fixup_main_from_path(data['init_main_from_path']) main_content = runpy.run_path(main_path,
main_content = runpy.run_path(main_path,

return _run_module_code(code, init_globals, run_name,
File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code
main_content = runpy.run_path(main_path, File "C:\Users\winado\Anaconda3\lib\multiprocessing\spawn.py", line 287, in _fixup_main_from_path

File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path

File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path

  File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code
  File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,
_run_code(code, mod_globals, init_globals,main_content = runpy.run_path(main_path,    _run_code(code, mod_globals, init_globals,

File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in _run_code
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path
exec(code, run_globals)
File "C:\Users\winado\Desktop\PICK\train.py", line 19, in
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code
return _run_module_code(code, init_globals, run_name,

main_content = runpy.run_path(main_path,

File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in _run_code
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path
return _run_module_code(code, init_globals, run_name,return _run_module_code(code, init_globals, run_name, File "C:\Users\winado\Anaconda3\lib\runpy.py", line 265, in run_path
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals,

exec(code, run_globals)  File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code

File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code
return _run_module_code(code, init_globals, run_name,return _run_module_code(code, init_globals, run_name, File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in _run_code
_run_code(code, mod_globals, init_globals,

  File "C:\Users\winado\Desktop\PICK\train.py", line 19, in <module>

exec(code, run_globals) File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 97, in _run_module_code
_run_code(code, mod_globals, init_globals, File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in _run_code
_run_code(code, mod_globals, init_globals,
from trainer import Trainer
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in _run_code
from trainer import Trainer
File "C:\Users\winado\Desktop\PICK\train.py", line 19, in 

    exec(code, run_globals)

exec(code, run_globals) run_code(code, mod_globals, init_globals, File "C:\Users\winado\Desktop\PICK\trainer_init.py", line 5, in
run_code(code, mod_globals, init_globals, File "C:\Users\winado\Desktop\PICK\trainer_init.py", line 5, in

File "C:\Users\winado\Desktop\PICK\train.py", line 19, in

from trainer import Trainer

File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in _run_code
File "C:\Users\winado\Desktop\PICK\train.py", line 19, in

File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in _run_code
File "C:\Users\winado\Anaconda3\lib\runpy.py", line 87, in run_code
from trainer import Trainer File "C:\Users\winado\Desktop\PICK\trainer_init.py", line 5, in

exec(code, run_globals)from trainer import Trainerexec(code, run_globals)exec(code, run_globals)
from .trainer import *

File "C:\Users\winado\Desktop\PICK\trainer_init_.py", line 5, in

File "C:\Users\winado\Desktop\PICK\train.py", line 19, in
from .trainer import *
File "C:\Users\winado\Desktop\PICK\train.py", line 19, in
File "C:\Users\winado\Desktop\PICK\trainer_init_.py", line 5, in
File "C:\Users\winado\Desktop\PICK\train.py", line 19, in

from .trainer import * File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in
from .trainer import * from trainer import Trainer File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in

from trainer import Trainerfrom .trainer import *
from trainer import Trainer
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in

File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in

File "C:\Users\winado\Desktop\PICK\trainer_init_.py", line 5, in
File "C:\Users\winado\Desktop\PICK\trainer_init_.py", line 5, in
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in
File "C:\Users\winado\Desktop\PICK\trainer_init_.py", line 5, in
from .trainer import *from .trainer import *from .trainer import *

File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 14, in
from utils.metrics import MetricTracker, SpanBasedF1MetricTrackerfrom utils.metrics import MetricTracker, SpanBasedF1MetricTrackerfrom utils.metrics import MetricTracker, SpanBasedF1MetricTrackerfrom utils.metrics import MetricTracker, SpanBasedF1MetricTrackerfrom utils.metrics import MetricTracker, SpanBasedF1MetricTracker from utils.metrics import MetricTracker, SpanBasedF1MetricTrackerfrom utils.metrics import MetricTracker, SpanBasedF1MetricTracker
File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in

File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in
from utils.metrics import MetricTracker, SpanBasedF1MetricTracker

File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in

File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in
File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in
File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in
File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in
File "C:\Users\winado\Desktop\PICK\utils\metrics.py", line 10, in
from .span_based_f1 import SpanBasedF1Measure from .span_based_f1 import SpanBasedF1Measure
from .span_based_f1 import SpanBasedF1Measurefrom .span_based_f1 import SpanBasedF1Measurefrom .span_based_f1 import SpanBasedF1Measurefrom .span_based_f1 import SpanBasedF1Measurefrom .span_based_f1 import SpanBasedF1Measure
from .span_based_f1 import SpanBasedF1Measure File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in

File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in

File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in
File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in
File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in
File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in
File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in
File "C:\Users\winado\Desktop\PICK\utils\span_based_f1.py", line 13, in
from allennlp.training.metrics.metric import Metricfrom allennlp.training.metrics.metric import Metricfrom allennlp.training.metrics.metric import Metricfrom allennlp.training.metrics.metric import Metricfrom allennlp.training.metrics.metric import Metricfrom allennlp.training.metrics.metric import Metricfrom allennlp.training.metrics.metric import Metricfrom allennlp.training.metrics.metric import Metric

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training_init_.py", line 1, in
from allennlp.training.checkpointer import Checkpointerfrom allennlp.training.checkpointer import Checkpointer from allennlp.training.checkpointer import Checkpointerfrom allennlp.training.checkpointer import Checkpointer

from allennlp.training.checkpointer import Checkpointer

from allennlp.training.checkpointer import Checkpointerfrom allennlp.training.checkpointer import Checkpointerfrom allennlp.training.checkpointer import Checkpointer File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in
from allennlp.training import util as training_util File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in

from allennlp.training import util as training_utilfrom allennlp.training import util as training_util
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in

from allennlp.training import util as training_utilfrom allennlp.training import util as training_util
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\checkpointer.py", line 14, in
from allennlp.training import util as training_util File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in
from allennlp.training import util as training_util
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in 

    from allennlp.data import Instance, Vocabulary, Batch

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in
from allennlp.training import util as training_util File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in
from allennlp.data import Instance, Vocabulary, Batchfrom allennlp.data import Instance, Vocabulary, Batch File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in 

from allennlp.data import Instance, Vocabulary, Batch

from allennlp.data import Instance, Vocabulary, Batch File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\training\util.py", line 19, in
from allennlp.data import Instance, Vocabulary, Batch
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in

from allennlp.data import Instance, Vocabulary, Batch
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in
from allennlp.data.dataloader import DataLoader, allennlp_collatefrom allennlp.data import Instance, Vocabulary, Batch File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in
from allennlp.data.dataloader import DataLoader, allennlp_collatefrom allennlp.data.dataloader import DataLoader, allennlp_collate

from allennlp.data.dataloader import DataLoader, allennlp_collate

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in
from allennlp.data.dataloader import DataLoader, allennlp_collate File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data_init_.py", line 1, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in
from allennlp.data.dataloader import DataLoader, allennlp_collate File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in 

  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in <module>

  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in <module>
from allennlp.data.dataloader import DataLoader, allennlp_collatefrom allennlp.data.dataloader import DataLoader, allennlp_collatefrom allennlp.data.instance import Instance  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in <module>
from allennlp.data.instance import Instancefrom allennlp.data.instance import Instance

from allennlp.data.instance import Instance

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\dataloader.py", line 9, in
from allennlp.data.instance import Instance File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in

from allennlp.data.instance import Instance File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in 

  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in <module>

    from allennlp.data.instance import Instance  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in <module>
from allennlp.data.instance import Instance  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in <module>
from allennlp.data.fields.field import DataArray, Fieldfrom allennlp.data.fields.field import DataArray, Field

from allennlp.data.fields.field import DataArray, Field
from allennlp.data.fields.field import DataArray, Field

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\instance.py", line 3, in
from allennlp.data.fields.field import DataArray, Field
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields_init_.py", line 15, in
from allennlp.data.fields.field import DataArray, Field File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields_init_.py", line 15, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields_init_.py", line 15, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields_init_.py", line 15, in 

    from allennlp.data.fields.field import DataArray, Field  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\__init__.py", line 15, in <module>
from allennlp.data.fields.field import DataArray, Field  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\__init__.py", line 15, in <module>
from allennlp.data.fields.namespace_swapping_field import NamespaceSwappingFieldfrom allennlp.data.fields.namespace_swapping_field import NamespaceSwappingField

from allennlp.data.fields.namespace_swapping_field import NamespaceSwappingField
from allennlp.data.fields.namespace_swapping_field import NamespaceSwappingField

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields_init_.py", line 15, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields_init_.py", line 15, in
from allennlp.data.fields.namespace_swapping_field import NamespaceSwappingField
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in
from allennlp.data.fields.namespace_swapping_field import NamespaceSwappingField File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in 

    from allennlp.data.tokenizers.token import Token  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in <module>
from allennlp.data.fields.namespace_swapping_field import NamespaceSwappingField  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in <module>
from allennlp.data.tokenizers.token import Tokenfrom allennlp.data.fields.namespace_swapping_field import NamespaceSwappingField

from allennlp.data.tokenizers.token import Token
from allennlp.data.tokenizers.token import Token
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers_init_.py", line 9, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers_init_.py", line 9, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in
from allennlp.data.tokenizers.token import Token

from allennlp.data.tokenizers.token import Token  File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\__init__.py", line 9, in <module>

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers_init_.py", line 9, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\fields\namespace_swapping_field.py", line 8, in
from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizer
from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers_init_.py", line 9, in
from allennlp.data.tokenizers.token import Token

from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizer
from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers_init_.py", line 9, in
from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizerfrom allennlp.data.tokenizers.token import Token File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in

File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers_init_.py", line 9, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in
from transformers import PreTrainedTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in
from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizerfrom transformers import PreTrainedTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers_init_.py", line 9, in 

from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizer

File "C:\Users\winado\Anaconda3\lib\site-packages\transformers_init_.py", line 99, in
from transformers import PreTrainedTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers_init_.py", line 46, in
from transformers import PreTrainedTokenizer from transformers import PreTrainedTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in

from transformers import PreTrainedTokenizer
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers_init_.py", line 23, in
from .pipelines import (from allennlp.data.tokenizers.pretrained_transformer_tokenizer import PretrainedTransformerTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\transformers_init_.py", line 46, in

File "C:\Users\winado\Anaconda3\lib\site-packages\transformers_init_.py", line 99, in
from .data import (from transformers import PreTrainedTokenizer

  File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\__init__.py", line 46, in <module>

File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\pipelines.py", line 36, in
File "C:\Users\winado\Anaconda3\lib\site-packages\allennlp\data\tokenizers\pretrained_transformer_tokenizer.py", line 5, in
from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfig from .data import ( File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data_init_.py", line 5, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers_init_.py", line 46, in
from .pipelines import (
from .data import (

  File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\configuration_albert.py", line 18, in <module>

  File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data\__init__.py", line 5, in <module>

from .tokenization_auto import AutoTokenizerfrom transformers import PreTrainedTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\pipelines.py", line 36, in
from .metrics import is_sklearn_available File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data_init_.py", line 5, in
from .data import (

from .configuration_utils import PretrainedConfig File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_auto.py", line 45, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers_init_.py", line 23, in
from .metrics import is_sklearn_available File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data\metrics_init_.py", line 18, in
from .tokenization_auto import AutoTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data_init_.py", line 5, in

from .metrics import is_sklearn_available

  File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\configuration_utils.py", line 25, in <module>

File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data\metrics_init_.py", line 18, in
from .configuration_albert import ALBERT_PRETRAINED_CONFIG_ARCHIVE_MAP, AlbertConfigfrom .tokenization_bart import BartTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_auto.py", line 45, in
from scipy.stats import pearsonr, spearmanrfrom .metrics import is_sklearn_available

File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data\metrics_init_.py", line 18, in

File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats_init_.py", line 391, in
from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_urlfrom .tokenization_bart import BartTokenizerfrom scipy.stats import pearsonr, spearmanr File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_bart.py", line 18, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\data\metrics_init_.py", line 18, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\configuration_albert.py", line 18, in

from scipy.stats import pearsonr, spearmanr File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_bart.py", line 18, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\file_utils.py", line 51, in
File "C:\Users\winado\Anaconda3\lib\site-packages\scipy_init_.py", line 132, in

from .stats import *from scipy.stats import pearsonr, spearmanrfrom .tokenization_roberta import RobertaTokenizerfrom .configuration_utils import PretrainedConfig

File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats_init_.py", line 391, in
from .tokenization_roberta import RobertaTokenizer File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\configuration_utils.py", line 25, in
File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats\stats.py", line 180, in
import tensorflow as tf File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats_init_.py", line 391, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_roberta.py", line 24, in

from scipy._lib import _pep440 from .file_utils import CONFIG_NAME, cached_path, hf_bucket_url, is_remote_urlfrom .stats import * File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_roberta.py", line 24, in

from .stats import * File "C:\Users\winado\Anaconda3\lib\site-packages\scipy_lib_pep440.py", line 285, in
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow_init_.py", line 41, in
from . import distributions File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\file_utils.py", line 51, in
from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats\stats.py", line 180, in
from .tokenization_gpt2 import GPT2Tokenizer, GPT2TokenizerFast
File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats\distributions.py", line 11, in 

class Version(_BaseVersion):

import tensorflow as tf  File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats\stats.py", line 180, in <module>
from tensorflow.python.tools import module_util as _module_util

File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_gpt2.py", line 23, in
File "C:\Users\winado\Anaconda3\lib\site-packages\transformers\tokenization_gpt2.py", line 23, in
from . import distributions
from . import _continuous_distns
File "C:\Users\winado\Anaconda3\lib\site-packages\scipy_lib_pep440.py", line 287, in Version

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow_init_.py", line 41, in

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python_init_.py", line 46, in
from . import distributions File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats\distributions.py", line 8, in
import regex as reimport regex as re File "", line 991, in _find_and_load

_regex = re.compile(

    from tensorflow.python.tools import module_util as _module_util  File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats\distributions.py", line 11, in <module>

File "", line 975, in _find_and_load_unlocked

File "C:\Users\winado\Anaconda3\lib\site-packages\regex_init_.py", line 1, in
File "C:\Users\winado\Anaconda3\lib\site-packages\regex_init_.py", line 1, in
from ._distn_infrastructure import (entropy, rv_discrete, rv_continuous,from tensorflow.python import data
File "", line 671, in _load_unlocked
File "C:\Users\winado\Anaconda3\lib\re.py", line 252, in compile

File "C:\Users\winado\Anaconda3\lib\site-packages\scipy\stats_distn_infrastructure.py", line 352, in
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python_init_.py", line 46, in
from . import _continuous_distns File "", line 839, in exec_module
from .regex import *
from .regex import *

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data_init_.py", line 25, in
File "C:\Users\winado\Anaconda3\lib\site-packages\regex\regex.py", line 419, in
File "", line 934, in get_code
return _compile(pattern, flags)
File "", line 991, in _find_and_load
from tensorflow.python import data
File "C:\Users\winado\Anaconda3\lib\site-packages\regex\regex.py", line 419, in
File "", line 1033, in get_data

exec('del ' + obj)from tensorflow.python.data import experimental File "C:\Users\winado\Anaconda3\lib\re.py", line 304, in _compile
File "", line 975, in _find_and_load_unlocked
import regex._regex_core as regex_core File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data_init.py", line 25, in
MemoryError

  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked

import regex._regex_core as regex_core MemoryError File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\experimental_init.py", line 97, in
File "C:\Users\winado\Anaconda3\lib\site-packages\regex_regex_core.py", line 21, in
p = sre_compile.compile(pattern, flags) File "", line 839, in exec_module

from tensorflow.python.data import experimental

from tensorflow.python.data.experimental import service    import regex._regex as _regex

ImportError: DLL load failed while importing _regex: The paging file is too small for this operation to complete.

File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 768, in compile
File "", line 991, in _find_and_load

File "", line 934, in get_code
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\experimental_init_.py", line 97, in
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\experimental\service_init_.py", line 353, in
File "", line 975, in _find_and_load_unlocked
File "", line 1033, in get_data
code = _code(p, flags) File "", line 671, in _load_unlocked
from tensorflow.python.data.experimental import service
MemoryError File "", line 839, in exec_module

File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 607, in code
from tensorflow.python.data.experimental.ops.data_service_ops import distribute
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\experimental\service_init.py", line 353, in
File "", line 934, in get_code

  File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\experimental\ops\data_service_ops.py", line 30, in <module>

File "", line 1033, in get_data
_compile(code, p.data, flags)MemoryErrorfrom tensorflow.python.data.experimental.ops.data_service_ops import distribute

  File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\experimental\ops\data_service_ops.py", line 30, in <module>

File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 156, in _compile
from tensorflow.python.data.ops import dataset_ops
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 39, in
from tensorflow.python.data.ops import dataset_ops_compile(code, av[2], flags)

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 39, in
File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 168, in _compile
from tensorflow.python.data.ops import iterator_ops
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 39, in
from tensorflow.python.data.ops import iterator_ops_compile(code, p, _combine_flags(flags, add_flags, del_flags))

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 39, in
File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 148, in _compile
from tensorflow.python.training.saver import BaseSaverBuilderfrom tensorflow.python.training.saver import BaseSaverBuilder

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 39, in
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\training\saver.py", line 53, in
_compile(code, av[2], flags)
File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 120, in _compile
from tensorflow.python.framework import meta_graphfrom tensorflow.python.training.saving import saveable_object_util

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\framework\meta_graph.py", line 39, in
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\training\saving\saveable_object_util.py", line 24, in
charset, hascased = _optimize_charset(av, iscased, tolower, fixes)
File "C:\Users\winado\Anaconda3\lib\sre_compile.py", line 320, in _optimize_charset
from tensorflow.python.framework import importer
from tensorflow.python.eager import def_function File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\framework\importer.py", line 28, in
charmap += b'\0' * 0xff00

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\eager\def_function.py", line 75, in
MemoryError
from tensorflow.python.framework import function
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\framework\function.py", line 37, in
from tensorflow.python.distribute.parallel_device import parallel_device
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\distribute\parallel_device\parallel_device.py", line 33, in
from tensorflow.python.ops import resource_variable_opsfrom tensorflow.python.tpu.ops import tpu_ops

File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\ops\resource_variable_ops.py", line 49, in
File "C:\Users\winado\Anaconda3\lib\site-packages\tensorflow\python\tpu\ops\tpu_ops.py", line 26, in
from tensorflow.python.ops import variables
File "", line 991, in _find_and_load
from tensorflow.python.ops import gen_tpu_ops
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 671, in _load_unlocked
File "", line 839, in exec_module
File "", line 839, in exec_module
File "", line 934, in get_code
File "", line 934, in get_code
File "", line 1033, in get_data
File "", line 1033, in get_data
MemoryError
MemoryError
Traceback (most recent call last):
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 761, in _try_get_data
data = self._data_queue.get(timeout=timeout)
File "C:\Users\winado\Anaconda3\lib\queue.py", line 178, in get
raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "train.py", line 167, in
entry_point(config)
File "train.py", line 131, in entry_point
main(config, local_master, logger if local_master else None)
File "train.py", line 79, in main
trainer.train()
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 135, in train
result_dict = self._train_epoch(epoch)
File "C:\Users\winado\Desktop\PICK\trainer\trainer.py", line 199, in _train_epoch
for step_idx, input_data_item in enumerate(self.data_loader):
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 345, in next
data = self._next_data()
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 841, in _next_data
idx, data = self._get_data()
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 798, in _get_data
success, data = self._try_get_data()
File "C:\Users\winado\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 774, in _try_get_data
raise RuntimeError('DataLoader worker (pid(s) {}) exited unexpectedly'.format(pids_str))
RuntimeError: DataLoader worker (pid(s) 21916, 28716, 45548, 29736, 15064, 26248, 14120, 20168) exited unexpectedly
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nadolsw