Background task not executed on high load

[razvanavram]

I have a simple function that inserts in the database (RethinkDB) a generated user.
The functions looks like this:

` @router.get('/api/v1/create_load_testing_user')
async def create_load_testing_user(request: Request, background_tasks: BackgroundTasks):

    api_user_1 = copy.deepcopy(default_user_model)
    api_user_1["uid"] = get_uuid_number()
    api_user_1["profile"]["name"] = "API USER 1"
    api_user_1["login"]["client_id"] = get_uuid_number()
    api_user_1["login"]["client_secret"] = get_uuid_number()
    api_user_1["is_performance_test_user"] = True

background_tasks.add_task(
    login_test_performance_insert,
    api_user_1,
    request
)

return api_user_1


async def login_test_performance_insert(api_user_1, request):
    await r.table("users").insert(api_user_1).run(await get_rethink_conn(request))
    return True

`

Now, i am doing a performance test with AB:
ab -n 8000 -c 1000 https://link

And i am getting:
Complete requests: 8000 Failed requests: 6294 (Connect: 0, Receive: 0, Length: 6294, Exceptions: 0) Non-2xx responses: 1706

This means that i just lost 1706 user entries in the db

What am i doing wrong?

I have a http middleware that acquires a db connection, from a pool, before executing the route and releases it after the function is completed

[euri10]

Any reason why you insert in db using a background task?
If not I'd try without , background tasks run in a seperate thread, you may exhaust them?

[razvanavram]

Because i want to return an answer faster.
Without the background tasks i would get around 250 req/sec and with backgroundtasks i get 550 req/sec (with 8000 users in db ).

[euri10]

Interesting, will try tomorrow to look at it, in the meantime what is ab? I just know wrk, but I guess it's similar? And how do you serve your app, gunicorn behind uvicorn or just uvicorn, well could you share a little bit more about those settings :) Le mar. 24 sept. 2019 à 9:15 PM, razvanavram <notifications@github.com> a écrit :

…

Because i want to return an answer faster. Without the background tasks i would get around 250 req/sec and with backgroundtasks i get 550 req/sec (with 8000 users in db ). — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#565>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAINSPV7XTPQJEFNDGUZH43QLJRMBANCNFSM4I2DSRUA> .

[razvanavram]

ab (apache benchmarking tool) is just like wrk.

I've just switched from flask (from tiangolo) to FastApi and now i have this problems so please bear with me :).

I am serving my app with gunicorn -k uvicorn.workers.UvicornWorker -c "$GUNICORN_CONF" "$APP_MODULE"

And my gunicorn_conf looks like this:
web_concurrency = 4
loglevel = use_loglevel
workers = web_concurrency
bind = use_bind
keepalive = 120
timeout = 1200
backlog = 2048
limit_request_field_size = 0
errorlog = "-"

Also i use use_bind = "unix:/tmp/uvicorn.sock"

[euri10]

ok, bear in mind I never ever used rethinkdb, here's what I did and how it went (good from what I can tell from ab results), I used only uvicorn to serve the app as you'll see below

1. I ran rethinkdb with docker run --name some-rethink -p 28015:28015 -d rethinkdb
2. I used this snippet, adapted from yours, there are discrepancies, maybe that explains why all requests end up fine. In particular I used a simple await to the connection in the background task and didn't use the request
3. run the script, it will only use uvircorn, I wanted to keep things simple and seperate eventual issues
4. run ab ab -n 8000 -c 1000 http://localhost:8000/create_load_testing_user

(rapidFastAPItest) ➜  rapidFastAPItest git:(master) ✗ ab -n 8000 -c 1000 http://localhost:8000/create_load_testing_user
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking localhost (be patient)
Completed 800 requests
Completed 1600 requests
Completed 2400 requests
Completed 3200 requests
Completed 4000 requests
Completed 4800 requests
Completed 5600 requests
Completed 6400 requests
Completed 7200 requests
Completed 8000 requests
Finished 8000 requests


Server Software:        uvicorn
Server Hostname:        localhost
Server Port:            8000

Document Path:          /create_load_testing_user
Document Length:        226 bytes

Concurrency Level:      1000
Time taken for tests:   21.235 seconds
Complete requests:      8000
Failed requests:        0
Total transferred:      2816000 bytes
HTML transferred:       1808000 bytes
Requests per second:    376.73 [#/sec] (mean)
Time per request:       2654.421 [ms] (mean)
Time per request:       2.654 [ms] (mean, across all concurrent requests)
Transfer rate:          129.50 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  865 1640.7     10    7145
Processing:    44  977 615.7    898    4716
Waiting:        3  819 603.0    735    4542
Total:         45 1842 1869.0   1037    9023

Percentage of the requests served within a certain time (ms)
  50%   1037
  66%   1734
  75%   1988
  80%   2864
  90%   4173
  95%   5521
  98%   8362
  99%   8368
 100%   9023 (longest request)

So the issue you face may come from how you handle the connection, would be interesting to see how you deal with it, again I'm a total rethinkdb newbie so I may be totally off

[euri10]

using gunicorn -k uvicorn.workers.UvicornWorker -c 565_gunicorn.conf "565_background_high_load:app" I get 1400 req/s without one failing
however I used bind = "0.0.0.0:8000" as you can see in conf used
what's the purpose of using a socket in your conf ?

[razvanavram]

Pfff. I will definitely try without a connection pool and spawn a new connection every time.
I will try your example (since it's a huge difference in the req/sec and no errors) and come back with the results.

The use of socket is for nginx communication. I use nginx because i need to redirect http to https and i didn't knew how to do it with FastApi. The traffic comes from a google load balancer that has the ssl cert.

Sidenote:
I use the http middleware to acquire a connection from the pool, execute the route function and after the function, i release the connection back to the pool. The db connection is then assigned to the request.state (eg. request.state.rethinkdb_connection) and accessed where i need it.

If i use the same approach with a new connection every time (each request) the background thread will throw an error saying that the connection is closed. And it's correct since after the route function i close the connection and the background thread runs after the request completed.

[sandys]

i would like to link my issue about production configuration of gunicorn here - #551

I'm very keen to figure out this tuning. Is it recommended to run gunicorn with threads ?

also - how do you create the connection pool in the middleware ? we are using sqlalchemy with postgresql, but a lot of the same things will apply.

The first place i learnt this from was the official sqlalchemy/postgresql starter for fastapi - https://github.com/tiangolo/full-stack-fastapi-postgresql/blob/b96d78db99d8c4f37f0df64df587ffebda4bfc56/%7B%7Bcookiecutter.project_slug%7D%7D/backend/app/app/main.py#L33

[razvanavram]

@sandys 'Is it recommended to run gunicorn with threads ?' - > i don't know what to say
'how do you create the connection pool in the middleware ? ' i am not creating the pool in the middleware. On the application startup i create the pool. In the middleware i just acquire a connection from the pool.

[razvanavram]

@euri10
I ran your test and i get (approx.) the same results.
I made some changes on my side and the problems with the dropped requests came from nginx with 2 timeouts.

Fixed that (dropped requests) but now the problem is that i am getting around 500 req/seq instead of 1300-1400 req/sec.

The only reason i use nginx is to redirect http to https. (running behind a google cloud loadbalancer)
Is there a solution to upgrade http to https using starlette/fastapi?

[euri10]

Is there a solution to upgrade http to https using starlette/fastapi?

not that I'm aware of, and nginx is supposed to be fastest than traefik which I currently use , but maybe others have better insights on that part

[razvanavram]

@euri10 well, thanks for the help

[sandys]

Have you guys tested haproxy ? Generally speaking it's faster (and simpler config than nginx)

…

On Wed, 25 Sep, 2019, 19:14 razvanavram, ***@***.***> wrote: @euri10 <https://github.com/euri10> well, thanks for the help — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#565>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAASYU3NX3G56SYBROZSDLDQLNTNFANCNFSM4I2DSRUA> .