Parralelization fail

[sshleifer]

Any idea how to fix? Has just started happening without changing code. Thought this was from running too many kernels at once but even if I run one kernel I get this error....

From kaggle script environment

WORBBAG_ITEM_DESC_PARAMS = {'hash_ngrams': 2, 'hash_ngrams_weights': [1.0, 1.0],
                            'hash_size': 2 ** 26, 'norm': 'l2', 'tf': 1.0, 'idf': None}
   wb = wordbatch.WordBatch(normalize_text, extractor=(WordBag, WORBBAG_ITEM_DESC_PARAMS),
                             procs=procs)
    wb.dictionary_freeze= True
    X_description = wb.fit_transform(full_df['item_description'])

2237.9s
477
Parallelization fail. Method: multiprocessing Task: <function batch_normalize_texts at 0x7f76ce912ea0>
Retrying, attempt: 1 timeout limit: 1200 seconds
2250.8s
478
Parallelization fail. Method: multiprocessing Task: <function batch_normalize_texts at 0x7f76ce912ea0>
Retrying, attempt: 2 timeout limit: 2400 seconds
2263.4s
479
Parallelization fail. Method: multiprocessing Task: <function batch_normalize_texts at 0x7f76ce912ea0>
Retrying, attempt: 3 timeout limit: 4800 seconds
2276.2s
480
Parallelization fail. Method: multiprocessing Task: <function batch_normalize_texts at 0x7f76ce912ea0>
Retrying, attempt: 4 timeout limit: 9600 seconds
2288.9s
481
Parallelization fail. Method: multiprocessing Task: <function batch_normalize_texts at 0x7f76ce912ea0>
Extract wordbags
2288.9s
482
Traceback (most recent call last):
  File "../src/script.py", line 573, in <module>
    sparse_mat = preprocess_for_fm(full_df)
  File "../src/script.py", line 552, in preprocess_for_fm
    X_description = wb.fit_transform(full_df['item_description'])
  File "/opt/conda/lib/python3.6/site-packages/wordbatch/wordbatch.py", line 230, in fit_transform
    return self.transform(texts, labels, extractor, cache_features, input_split)
  File "/opt/conda/lib/python3.6/site-packages/wordbatch/wordbatch.py", line 242, in transform
    texts= extractor.transform(texts, input_split= True, merge_output= True)
  File "wordbatch/extractors/extractors.pyx", line 185, in wordbatch.extractors.extractors.WordBag.transform
  File "/opt/conda/lib/python3.6/site-packages/wordbatch/wordbatch.py", line 305, in parallelize_batches
    paral_params=  [[data_batch]+ args for data_batch in data]
TypeError: 'NoneType' object is not iterable
2288.9s
483
2288.9s
484
Failed. Exited with code 1.

[sshleifer]

Also the parrelization fail warnings are 13 seconds apart but cite timeout limits that are 1200/2400/4800 seconds apart?

[anttttti]

Usually this is something wrong with the input. Try adding method="serial" to the wb initializer. This will run calls serially without multiprocessing, so you will see where the code crashes.

The fail warnings come as soon as the parallelization fails. The timeout limits are doubled on each attempt.

[sshleifer]

thanks that helped me figure it out!

…

On Sun, Feb 11, 2018 at 3:08 AM anttttti ***@***.***> wrote: Usually this is something wrong with the input. Try adding method="serial" to the wb initializer. This will run call serially without multiprocessing, so you will see where the code crashes. The fail warnings come as soon as the parallelization fails. The timeout limits are doubled on each attempt. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#9 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AFw9Yd8-eym89CbnCM2o4h5sxaOz12hKks5tTp_zgaJpZM4SBKVB> .