-
Notifications
You must be signed in to change notification settings - Fork 6
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Training crashes at end #4
Comments
Update: I spoke a little too soon on the blanks issue. There are still a very small number of blanks in the output file. However, it's a major improvement. Instead of nearly half of the outputs being blank, out of 1374971 input lines, only 211 resulted in blank values. |
yes we faced a similar issue. see pytorch/text#337 I was able to train a big model without any issue but obviously in some situation this is happening. |
@devinbostIL can you pull and try again ? |
Might be working now :)
(whereas before I got |
Update: seems like saving is working without crashing for me now. Not sure if I was encountering the same issue as @devinbostIL ?
|
closing this, reopen it if needed |
It's running to the end without crashing now. |
Make comments compatible with doc format
* wip simplify shards stuff * some cleanup * rename shard to corpus * some cleanup * multiple producers, move transform up in code execution * apply transform properly in verbose mode * do not use cycle, which causes memory leak * make one queue per gpu per producer, to preserve reproducibility * enable bpe dropout for pyonmttok, v>1.19, fix inverted dropout value * terminate producers properly when training processes are done * single queue per producer/consumer couple * revert semaphore size change
* bin/train_dynamicdata.py * opts: --data_config and --bucket_size * train_single.main_dynamicdata * dynamicdata package * broken * still broken: not enough to get rid of immediate generator * build_data_loader in producer * nfeats 0, not 1 * fixes to mismatching batch structure * looks weird reporting while filling up queue * bin/preprocess_dynamicdata.py * enable PrefixTransform to work with only src * WIP: translator for dynamicdata * WIP: opened too early * dynamicdata runs * removed obsolete function * restore mixing weight schedule counter * pop all expired mixing weights at once * compressed inputs * allow missing transforms during sharding * fail fast on missing files * bad cleverness in vocab paths * repr for transforms * SentencepieceTransform * send cpu tensors from producer to consumer The data loader cannot access the gpu when CUDA Compute Mode is set to Exclusive_Process. Trying to do so results in "CUDA error: all CUDA-capable devices are busy or unavailable". As the training process has the gpu, it is responsible for calling torch.device to send data from cpu to gpu. * torch.device moved to wrong place. Also missing from validation. * prints to logging * access subword vocab from transforms. SwitchOut * WbNoiseTransform * multiple transform pipelines with share_inputs * reverse: flips src and trg for backtranslation * saving and loading data_loader_step * pre-de-tokenization * MorfessorEmTabooPeturbedTransform * logging (on and off) * parameter seg_n_samples * protect from clobbering * extra_prefix * comment * bin/frankenstein.py * prune matching keys when config doesn't match * insertion transform * order of _inputs shouldn't matter * Sentencepiece sampling params * need to store in pickle * started writing dynamicdata documentation * critique of the current dataloader * minor * concepts * usage * dyndata.md * updated fig * better naming * pep8 * translate opt * re.groups should not be changed to tasks * rename group to task in translator * replaced readme with dynamicdata version * dynamicdata requirements as optional * DeterministicSegmentationTransform * WIP: bin/debug_dynamicdata.py * bin/debug_dynamicdata.py * DeterministicSegmentationTransform * sanity checks to SentencepieceTransform * update README.md * formatting * template_dynamicdata.py * Cleaner structure of sharded directory. Separate shards from vocabs/transforms * pep8 * completed some 2dos * Automatically determine input corpus sizes * use line counts to ensure shards of even size * update README.md * don't die on blanks * spm_to_vocab.py, minor bugfixes * Missing subword vocab conversion step from SentencePiece usage. * clarified that meta.train.vocab_path points to subword vocab * morfessor expected counts to vocab * use Morfessor EM+Prune models directly as vocab * pretokenize=True silently ignored * documentation: pretokenization, exclusive_process * arxiv link * documentation: dampening, pretokenizing test set * mention BART * minor * note about BPE-Dropout * reverse README * fix multiprocess & pooling * some cleaning * merge train_single * fix gpu_rank for cpu * add adapted dynamic data iterate mecanism * some cleaning * clean train script * remove deprecated code * remove deprecated code, patch 2 * fix multiprocessing not pickle generator for train dynamic * fix some mentioned issues & update train config example file * more transforms * drop sharding, iterate directely on full corpus * transform composite and statistics * add some documentation * improve transform perf. relate to sampling * add ONMTTokenizer Transform & use new interface of sentencepiece * MixingStrategy for DatasetIter to make it interchangeable * fix relate to valid_iter * fix translate with dynamic trained model * misc fix * add BPE * save dynamic sample if verbose * [WIP] Some adaptations to the dynamic data pipeline (#4) * wip simplify shards stuff * some cleanup * rename shard to corpus * some cleanup * multiple producers, move transform up in code execution * apply transform properly in verbose mode * do not use cycle, which causes memory leak * make one queue per gpu per producer, to preserve reproducibility * enable bpe dropout for pyonmttok, v>1.19, fix inverted dropout value * terminate producers properly when training processes are done * single queue per producer/consumer couple * revert semaphore size change * make transforms a folder * add bart noise as transform * register transform to support extension (Ubiqus#6) * fix to BART * remove preprocess from dynamic (Ubiqus#8) * do not make links, make counters * add initial build_vocab script * merge preprocess into train * compute index with stride and offset * remove preprocess & some cleaning Co-authored-by: François Hernandez <francois.hernandez.fh@gmail.com> * enable transforms behavior change between train/valid * fix minor issue with bart * fix reproducibility for transforms (Ubiqus#9) * update doc & config example Co-authored-by: Stig-Arne Gronroos <stig-arne.gronroos@aalto.fi> Co-authored-by: François Hernandez <francois.hernandez.fh@gmail.com>
I setup a conda environment with PyTorch 0.4 and installed this fork of OpenNMT-py, as I mentioned in the original issue I had posted here: OpenNMT#743
I then ran:
python preprocess.py -train_src C:\src\torchevere-offensive-classifier\training\character\train_src.txt -train_tgt C:\src\torchevere-offensive-classifier\training\character\train_dst.txt -valid_src C:\src\torchevere-offensive-classifier\training\character\val_src.txt -valid_tgt C:\src\torchevere-offensive-classifier\training\character\val_dst.txt -save_data data/character/tc-offense-classifier-character_v3 -src_seq_length 5000 -tgt_seq_length 5000
python train.py -data data/character/tc-offense-classifier-character_v3 -save_model tc-offense-classifier-character_v3 -gpuid 0 -layers 3 -learning_rate_decay 0.99 -train_steps 10000 -rnn_size 500
Everything ran smoothly until it crashed at the end.
Here's the output:
Luckily, there was still a model file that was produced during the middle of the training (half way through the specified number of training steps), and I was able to use that with
translate.py
. And thankfully, it solved the issue with having blanks in the output file, as I mentioned in that original issue (OpenNMT#743).However, it's still an issue to have the training crash at the end of the process, so I'm reporting this bug.
The text was updated successfully, but these errors were encountered: