Supervised training gets stuck after training is 100% complete

[mrxiaohe]

I've been trying to use fastText in Windows command prompt.

D:\fastText>fasttext supervised -input D:\ClassificationTesting\docs_labels_fasttext_train.txt -output D:\ClassificationTesting\model_classificationtesting -epoch 10 -lr 1.0 -wordNgrams 2 -thread 40 -
Read 915M words
Number of words:  3873785
Number of labels: 2
Progress: 100.0% words/sec/thread:  983396 lr:  0.000000 loss:  0.203929 ETA:   0h 0m

Even though training is 100% complete as shown above, fastText doesn't actually terminate and doesn't return to the command prompt (i.e., D:\fastText>), and the task manager shows that fasttext is still running (see screenshot below).

This problem seems to go away when I run fastText without the wordNgrams option. I wonder what may have caused this issue.

[stas00]

I have a similar issue when used with python:

problem:
fastText gets stuck in 100% CPU spinning, even though it allows the rest of the code to continue to run when executed inside jupyter notebook:

installed fasttext:
!pip install git+https://github.com/facebookresearch/fastText.git

to reproduce:
first you will need to get https://s3-us-west-1.amazonaws.com/fasttext-vectors/wiki.en.zip
and unzip it.

then the code using it:

import fastText as ft
en_vecs = ft.load_model('wiki.en.bin')
vecd = {w:en_vecs.get_word_vector(w) for w in en_vecs.get_words()}
print("Hello") # this gets executed!

the control is returned to the notebook, but CPU is spinning at 100%

[stas00]

a month passed, not even an ack has been received. So we can't use this implementation of fastText. Will look for an alternative implementation instead.

[Celebio]

Hi @mrxiaohe , @stas00 ,
Thank you for reporting the issue.
@stas00 , are you also working on a windows environment?

Regards,
Onur

[stas00]

@Celebio, I'm on Ubuntu 18.04.

[Celebio]

@stas00 ,
I am not able to reproduce the testcase. The cpu indeed continues to spin at 100%, the operation takes a while, but I don't have the control back to the notebook. Any print function is executed only after.

Could you please try the following and tell me if you observe something similar:

vecd = {w:en_vecs.get_word_vector(w) for w in en_vecs.get_words()[:10000]}

(and various limits like 100000, 200000, so on).

By the way, if you don't care about the model itself, you can use this snippet to load the vectors with vecd = load_vectors("wiki.en.vec"). (It still takes some time for this particular file, however).

Regards,
Onur

[Celebio]

@mrxiaohe ,
Could you please send me your docs_labels_fasttext_train.txt file?

Regards,
Onur

[stas00]

@stas00 ,
I am not able to reproduce the testcase. The cpu indeed continues to spin at 100%, the operation takes a while, but I don't have the control back to the notebook. Any print function is executed only after.

Could you please try the following and tell me if you observe something similar:

vecd = {w:en_vecs.get_word_vector(w) for w in en_vecs.get_words()[:10000]}

(and various limits like 100000, 200000, so on).

Thank you for suggesting to try with small numbers. I now have a better picture of what's happening on the user side.

If I use a small number, say 10, it returns right away and there is no CPU spinning. If I use 1000 it takes a few secs to return the control to the notebook and CPU is spinning for a few extra seconds after even though the notebook is idle.

If I use a bigger number say 20,000 - it takes perhaps 5 secs to give the control back to the notebook, and then some 30 secs of the CPU spinning at 100%. The bigger the number the longer the CPU is spinning after the cell is done executing. I haven't measured it, but with 10,000,000+ it's probably quite a few hours.

So, now we know that the process isn't really a runaway process - it's "temporarily" run-away and will recover in some hours. Which clearly indicates some bug or a strange behavior, isn't it?

Why is this happening? If the python one-liner finished executing, why does it take so much longer for the ft sub-process (or is it a thread?) to complete - what is it doing with the CPU after it finished executing?

[Celebio]

Hi @stas00 ,
I think the asynchronous nature of what you are observing comes from the way Jupyter handles connection between the notebook ui and it's kernel.

Are you able to reproduce this behaviour in a regular python console?

[stas00]

ipython's behavior of executing this code is dramatically different as compared to jupyter. It loads 10M records in under a minute and with hardly any CPU consumed (mainly disk io) and no CPU is consumed after the control is returned to ipython.

It almost feels like jupyter notebook is re-copying all that data somewhere for who knows what reason after it finished running the code, and it's not IO-bound, but CPU-bound.

In any case, thank you for helping me identify that this strange runaway-like process behavior has to do with some jupyter notebook's internal workings and not a bug in your software.

Unfortunately, the problem remains is that this software is not suitable for jupyter notebook environment :(

Thank you, @Celebio.

[dontloo]

I get the same problem on both windows and ubuntu, and weirdly I can use the "hs" option together with "wordNgrams" but not the ''ns" option