spaCy multiprocessing ('n_process') makes Docker image exit

[dave-espinosa]

Hello everyone,

Long story short: I need to apply some NER on several thousands of documents. I have decided to go with the "en_core_web_lg" model. I want to increase the NER processing speed as much as possible, so the speed FAQ has been already considered. Two of the main suggestions there, require the developer to use nlp.pipe and fine-tune the n_process argument in it, which I have already done, to obtain something like this:

import spacy
import os
nlp = spacy.load("en_core_web_lg")
raw_skill_list = [list(doc.ents) for doc in nlp.pipe(text_list, batch_size=17, n_process=os.cpu_count()-1)]
# NOTES:
# 1) `text_list` contains a long, long list of documents
# 2) The original code includes the disabling of some pipeline components, I have done that, but for the sake of this issue, it is not important

The results look fine so far, at least in the tests ran in my computer. To speed things up even more, and since my company works with GCP, I was planing to create a Docker image with the previous code in it, to then deploy it in several endpoints. The goal is of course split the 'text_list' in 'n' chunks, where n = number of endpoints. The problem that I have right now is that when n_process = 1, the Docker image manages to normally process all the texts, batch after batch (in the following toy example, 2 batches of 100 texts were used):

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
INFO:     172.17.0.1:35560 - "GET / HTTP/1.1" 200 OK
INFO:     172.17.0.1:35560 - "GET /docs HTTP/1.1" 200 OK
INFO:     172.17.0.1:35560 - "GET /openapi.json HTTP/1.1" 200 OK
Skills extracted from 100 texts, in 0.28 seconds
Skills cleaned from 100 lists, in 0.00 seconds
Skills refined from 100 lists, in 1.29 seconds
INFO:     172.17.0.1:35564 - "POST /eng/plain/ HTTP/1.1" 200 OK
Skills extracted from 100 texts, in 0.28 seconds
Skills cleaned from 100 lists, in 0.00 seconds
Skills refined from 100 lists, in 1.29 seconds
INFO:     172.17.0.1:35564 - "POST /eng/plain/ HTTP/1.1" 200 OK

BUT if n_process > 1, then the image just processes the first batch, and then exits:

INFO:     Started server process [1]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8080 (Press CTRL+C to quit)
Skills extracted from 100 texts, in 0.21 seconds
Skills cleaned from 100 lists, in 0.00 seconds
Skills refined from 100 lists, in 1.32 seconds
INFO:     172.17.0.1:35792 - "POST /eng/plain/ HTTP/1.1" 200 OK
INFO:     Shutting down
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [1]

After some research in Google, even when "one process per Docker image" is usually recommended, there are ways to achieve multiprocessing. However and quite honestly I don't know how to adapt those suggestions for spaCy's nlp.pipe.

Have you had any experience implementing something similar to this?

PS.: I discarded this recommendation because my company's GCP architecture is better oriented to deal with Docker images.

Thank you.

[pmbaumgartner]

How many workers are you launching uvicorn with? I'm guessing there's a conflict there with the workers & n_process.

Is it possible to share the Dockerfile you're using?

[dave-espinosa]

Hello @pmbaumgartner ,

Regarding some of your queries:

Are you working on a Mac for your testing?

No, here are the specs of the machine I am currently working with:

• Processor: Intel(R) Core(TM) i5-8259U CPU @ 2.30GHz 2.30 GHz (8 CPU cores)
• Installed RAM: 32.0 GB (31.9 GB utilizable)
• System type: 64 bits operating system, x64 processor
• Operating system: Windows 10 Pro
• Version: 21H2
• OS compilation: 19044.1466
• Experience: Windows Feature Experience Pack 120.2212.3920.0

Unfortunately, since I am working remotely, I have only that machine "ready to work", however I am doing my research, to know how to use a GCP VM to deal with Docker as well as GitHub repos, and check if I can use its power to do my tests (as far as I know, those machines use Ubuntu as OS). This might take time from my side, however some colleague(s) will be working next week on this issue, so expect new attendants to this thread 😁.

Those incremental improvements from my benchmark make me think the optimal solution for your system might be to fine-tune the resources per-container and set n_process=2 for each of them (rather than one big box with n_process>>1). But again, this isn't my area of expertise so there may be something even better.

My colleague(s) who will be working next week on this research, already has(have) the link to this thread, so (t)he(y) will probably include his(their) results and see if we can notice any improvement from our side. And hopefully, we reach some new discovery on this regard. Granted, they will be running those tests from a linux-based OS, as per your suggestion.

Best regards, and hope to let you know what happened soon.

Thank you very much,

[pmbaumgartner]

@dave-espinosa The more the merrier! Looking forward to figuring things out.

One additional note: I neglected to mention my implication about multiprocessing being slow on macOS also applies to Windows, so I'm not yet convinced we can rule that out as the cause of the local slowdown.

[adestevez]

Hi @dave-espinosa @pmbaumgartner

I have performed the experiments as mentioned in the corresponding thread. However, prior to that neatly on an "Ubuntu 20.04.3 LTS OS, 64 bits, RAM 7,6 GiB, Intel® Core™ i7-8565U CPU @ 1.80GHz × 8, Mesa Intel® UHD Graphics 620 (WHL GT2), ", the results were as follows:

• Skills extracted from 10 texts, in 0.35 seconds
• Skills extracted from 100 texts, in 3.25 seconds
• Skills extracted from 1000 texts, in 33.98 seconds

The results obtained according to the comments posted in the thread, for 4 cores 0.59 seconds, while for 8 cores a time of 0.93 seconds.

It is important to mention that the workers were not used with the experiments since they initially worsened the experiments.

[dave-espinosa]

Hello @adestevez ,

Thank you very much for your experiments.

@pmbaumgartner , in general terms, the following insights can be extracted from Alexis' experiments:

• spaCy:
  • It seems that a linux-based OS does provide a boost in speed on the NER pipeline. But if this is the case, shouldn't some warning be present in spaCy's documentation about this, so the developer is aware that there could be time differences in speed tests, if the OS is changed (as we have discovered in this thread)?
• Others:
  • Not including the workers parameter (uvicorn) provides unconclusive / unstable behaviours when used with spaCy pipeline, if n_process>1. In Windows for instance, it caused to process only the first text batch and then exist the Docker image; in Ubuntu it causes to make pipeline even slower than not using it.

I fear we are reaching the limits of spaCy's performance, but the explanation as to why is this happening, is getting darker & spookier 👻😨... Just joking here 😉, sorry!

If you have some other clue, or a different result from some experiment on your side, please let us know.

Thank you!

[pmbaumgartner]

Realized I had replied at the top level and not in thread, apologies!

#10087 (comment)

[pmbaumgartner]

@dave-espinosa That all makes sense to me, thanks for summarizing.

We do have a note in the docs about the behavior, but it's not explicitly called out. We do include a link that I referenced throughout to explain what's going on.

Not including the workers parameter (uvicorn) provides unconclusive / unstable behaviours when used with spaCy pipeline, if n_process>1. In Windows for instance, it caused to process only the first text batch and then exist the Docker image; in Ubuntu it causes to make pipeline even slower than not using it.

The issue here is that workers and n_process both use multiprocessing. For example, If you have workers=4, that's 4 processes that you're running. Inside each of those workers, if you've set n_process to anything other than 1, that worker is also going to start additional subprocesses. Quickly you'll see this adds up to significant overhead as you now have uvicorn managing workers managing subprocesses.

In addition, it's easy to run out of memory in these scenarios, so you might want to look into that. The docker defaults are pretty low for memory use and typically when RAM is the issue you'll see the exit behavior like you have.

[dave-espinosa]

Thank you very much for your quick response @pmbaumgartner ,

Understood. I think we have got to a dead end here. There could be some space for improvement, but that would imply modifying current's spaCy source code to get full advantage of multiprocessing Python library (we will try to include those experiments, but they won't be a priority as of now). To wrap up, we will try to stick to the safe side, i.e.:

• Keep n_process=1 and tune speed only using batch_size.
• Try to use a linux-based OS for the endpoints.
• Split in as much endpoints as possible, to increase the NER processing speed even more.

Let me know if some other suggestion is being left-out in my checklist.

Thank you very much!