[11.0] Truncated/malformed long responses in multiprocessing mode with Python 3.5 #20158
Comments
|
Debian 9.2 package install has the same problem |
|
I'm having the same issue on Debian 9.2 with the following setup: Base Debian 9.2 install Additional deb packages installed: Plus these from pip3: I have also tried git cloning the source and using pip3 to install dependencies from requirements.txt with the same results. The workers don't want to work ;-) |
|
Also digging further there are missing js and css files which I assume are causing the UI problem http://localhost:8069/web/content/290-8bbd54f/web.assets_backend.js And if I comment out workers Odoo 11 behaves as expected. |
|
@Yenthe666, I can confirm this, I closed a duplicated issue which I made with more details and using the information here #20176 Install Odoo with this specifications:
Current behavior:20:59:27.057 (index):37 GET https://v11.iterativo.do/web/content/240-ba36516/web.assets_backend.0.css net::ERR_INCOMPLETE_CHUNKED_ENCODING
20:59:27.289 (index):38 GET https://v11.iterativo.do/web/content/241-ba36516/web.assets_backend.1.css net::ERR_INCOMPLETE_CHUNKED_ENCODING
20:59:27.311 (index):35 GET https://v11.iterativo.do/web/content/239-0a9c277/web.assets_common.0.css net::ERR_INCOMPLETE_CHUNKED_ENCODING
20:59:27.435 (index):47 GET https://v11.iterativo.do/web/content/245-ba36516/web.assets_backend.js net::ERR_INCOMPLETE_CHUNKED_ENCODING
20:59:27.491 (index):45 GET https://v11.iterativo.do/web/content/244-0a9c277/web.assets_common.js net::ERR_INCOMPLETE_CHUNKED_ENCODING
20:59:27.675 (index):49 GET https://v11.iterativo.do/web/content/246-ab351a2/web_editor.summernote.js net::ERR_INCOMPLETE_CHUNKED_ENCODING
20:59:27.676 (index):51 GET https://v11.iterativo.do/web/content/247-a129a5f/web_editor.assets_editor.js net::ERR_INCOMPLETE_CHUNKED_ENCODING
20:59:27.684 (index):61 Uncaught TypeError: odoo.define is not a function
at (index):61Debug LOG 2017-10-16 00:59:30,500 364 INFO pruebas odoo.modules.loading: Modules loaded.
2017-10-16 00:59:30,502 364 INFO pruebas odoo.addons.base.ir.ir_http: Generating routing map
2017-10-16 00:59:30,512 363 INFO pruebas odoo.modules.registry: At least one model cache has been invalidated, signaling through the database.
2017-10-16 00:59:30,526 359 INFO pruebas odoo.modules.loading: Modules loaded.
2017-10-16 00:59:30,527 363 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/241-ba36516/web.assets_backend.1.css HTTP/1.0" 200 -
2017-10-16 00:59:30,531 364 INFO pruebas odoo.modules.registry: At least one model cache has been invalidated, signaling through the database.
2017-10-16 00:59:30,532 364 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/239-0a9c277/web.assets_common.0.css HTTP/1.0" 200 -
2017-10-16 00:59:30,533 359 INFO pruebas odoo.addons.base.ir.ir_http: Generating routing map
2017-10-16 00:59:30,538 366 INFO pruebas odoo.modules.loading: Modules loaded.
2017-10-16 00:59:30,540 366 INFO pruebas odoo.addons.base.ir.ir_http: Generating routing map
2017-10-16 00:59:30,596 365 INFO pruebas odoo.modules.loading: Modules loaded.
2017-10-16 00:59:30,599 365 DEBUG pruebas odoo.modules.registry: Multiprocess signaling check: [Registry - 2 -> 2] [Cache - 3 -> 5]
2017-10-16 00:59:30,599 365 INFO pruebas odoo.modules.registry: Invalidating all model caches after database signaling.
2017-10-16 00:59:30,602 365 INFO pruebas odoo.addons.base.ir.ir_http: Generating routing map
2017-10-16 00:59:30,629 363 DEBUG pruebas odoo.modules.registry: Multiprocess signaling check: [Registry - 2 -> 2] [Cache - 4 -> 5]
2017-10-16 00:59:30,629 363 INFO pruebas odoo.modules.registry: Invalidating all model caches after database signaling.
2017-10-16 00:59:30,640 363 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/245-ba36516/web.assets_backend.js HTTP/1.0" 200 -
2017-10-16 00:59:30,700 366 INFO pruebas odoo.modules.registry: At least one model cache has been invalidated, signaling through the database.
2017-10-16 00:59:30,703 366 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/242-ab351a2/web_editor.summernote.0.css HTTP/1.0" 200 -
2017-10-16 00:59:30,712 359 INFO pruebas odoo.modules.registry: At least one model cache has been invalidated, signaling through the database.
2017-10-16 00:59:30,713 359 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/244-0a9c277/web.assets_common.js HTTP/1.0" 200 -
2017-10-16 00:59:30,724 365 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/243-a129a5f/web_editor.assets_editor.0.css HTTP/1.0" 200 -
2017-10-16 00:59:30,846 366 DEBUG pruebas odoo.modules.registry: Multiprocess signaling check: [Registry - 2 -> 2] [Cache - 6 -> 7]
2017-10-16 00:59:30,846 366 INFO pruebas odoo.modules.registry: Invalidating all model caches after database signaling.
2017-10-16 00:59:30,850 366 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/246-ab351a2/web_editor.summernote.js HTTP/1.0" 200 -
2017-10-16 00:59:30,875 364 DEBUG pruebas odoo.modules.registry: Multiprocess signaling check: [Registry - 2 -> 2] [Cache - 5 -> 7]
2017-10-16 00:59:30,875 364 INFO pruebas odoo.modules.registry: Invalidating all model caches after database signaling.
2017-10-16 00:59:30,879 364 INFO pruebas werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:30] "GET /web/content/247-a129a5f/web_editor.assets_editor.js HTTP/1.0" 200 -
2017-10-16 00:59:31,489 365 INFO ? werkzeug: 192.168.2.201 - - [16/Oct/2017 00:59:31] "GET /web_enterprise/static/src/img/mobile-icons/android-192x192.png HTTP/1.0" 200 -In Console with debug assets 21:08:23.786 ?debug=assets:70 GET https://v11.iterativo.do/web/static/src/less/web.assets_backend/import_bootstrap.less.css net::ERR_INCOMPLETE_CHUNKED_ENCODING |
|
This one could also be related #20175 |
|
Ticket 777033 |
|
I encountered the same issue. The weird thing is that if you connect to the longpolling port |
|
I ran into this also while testing with the odoo:11.0 docker image. The problem seems to be that the assets are not fetched completely, i.e. they are truncated. When running it behind nginx this error is shown: It seems the workers close the connection before sending all data. |
|
So… according to research by @odony and @julienlegros it seems to be an issue specifically on Python 3.5 on Linux, above a certain quantity of data sent over a non-blocking socket, so:
|
|
The issue is likely https://bugs.python.org/issue24291, it is indeed an issue on non-blocking sockets, fixed in 3.6, it can apparently happen on any platform but it's going to depend when the platform's sockets do partial writes (so probably depend on the kernel's buffers & tuning & all). |
|
You can try 253d49f7a404ab5c889c90446bf6e87efd4ce026 as a possible fix, but it is not fully validated yet. |
|
@gustavovalverde #20175 looks unrelated |
|
I'm wondering, based on @xmo-odoo sentence:
Could containerization be a root cause of this issue? As I'm using LXC containers for Odoo and #20158 (comment) is using Docker; maybe the kernel behavior (limitations) could also trigger this. |
|
@odony, patch 253d49f works 👍 . @gustavovalverde it seems to be an issue with python's socket library as it was referenced in the commit |
|
Thanks for the feedback! 253d49f has been merged in 11.0 at d9b721c. Hopefully it should have no adverse effect. I did what I could to limit the effects to the specific combination of Python 3.5 and multi-process mode, which are known required conditions to trigger the bug. (See the commit message of d9b721c for more details) |
odony
changed the title
fw-bot
pushed a commit
to odoo-dev/odoo
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: odoo#20158 X-original-commit: d78ea12
fw-bot
pushed a commit
to odoo-dev/odoo
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: odoo#20158 X-original-commit: d78ea12
fw-bot
pushed a commit
to odoo-dev/odoo
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: odoo#20158 X-original-commit: d78ea12
This was referenced May 27, 2020
fw-bot
pushed a commit
to odoo-dev/odoo
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: odoo#20158 X-original-commit: d78ea12
fw-bot
pushed a commit
to odoo-dev/odoo
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: odoo#20158 X-original-commit: d78ea12
robodoo
pushed a commit
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes #51978 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
robodoo
pushed a commit
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes #51974 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
robodoo
pushed a commit
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes #51971 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
robodoo
pushed a commit
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes #51968 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
robodoo
pushed a commit
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes #51965 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
robodoo
pushed a commit
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes #51982 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
robodoo
pushed a commit
that referenced
this issue
May 27, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes #51972 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
EditThis seems to be fixed with this PR: #51824 (Thanks @odony!) To summarize: It's indeed recommended to perform response buffering at the reverse proxy level (as described in @ejprice's comment) but the socket timeout environment variable Here Oliviers comment on it:
Would be great if this was mentioned in the deployment docs. Original comment@odony This issue is still present in the latest version of Odoo v14. I debugged it for three full days until I figured out that it's due to the The environmentOdoo version: v14.0 release 20201123 (CE and EE) [options]
db_maxconn = 128
db_name = my-db-name
dbfilter = ^my-db-name$
limit_time_cpu = 600
limit_time_real = 1800
proxy_mode = True
workers = 3Reproduction steps
curl 'http://<IP address or URL>/path/to/the/web.assets_backend.js' \
-H 'Cookie: session_id=2a572f8fba3d721bdbb18a5de6cd0536d5f8e3f5' \
--limit-rate 10K | tail -n 2
The solution suggested by @ejprice is working but only a workaround and doesn't fix the Odoo issue. Anyway, thanks for it! |
jvandri
pushed a commit
to Niboo/odoo
that referenced
this issue
Dec 10, 2020
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: odoo#20158 closes odoo#51824 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
AdriaGForgeFlow
pushed a commit
to ForgeFlow/odoo
that referenced
this issue
Jan 28, 2021
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: odoo#20158 closes odoo#51968 X-original-commit: d78ea12 Signed-off-by: Olivier Dony (odo) <odo@openerp.com>
BT-tbaechle
referenced
this issue
in brain-tec/odoo
Jul 13, 2022
As indicated in the comment, it's much preferred to perform response buffering at the reverse proxy level than to increase the socket timeout. It will free up HTTP workers for other requests faster, while the proxy does the work of buffering the stream on disk as needed. /!\ The timeout is also used to protect from accidental DoS effects in situations of low worker availability, due to idle connections caused e.g. by wkhtmltopdf's connection pooling. Setting a high timeout will make the protection less effective, so ensuring you have enough free HTTP workers at all times becomes critical. In our tests with nginx's defaut buffering on a typical hardware with SSD storage, buffering up to 1GB responses did not require any change of the socket timeout on the Odoo side, though your mileage may vary. See also nginx's `proxy_buffering` and `proxy_max_temp_file_size` config directives. OPW-2247730 See also: #20158 closes odoo#51824 Signed-off-by: Olivier Dony (odo) <odo@openerp.com> (cherry picked from commit d78ea12)
This was referenced Jul 13, 2022
Merged
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Impacted versions:
11.0 community
ubuntu 17.04 package install
Steps to reproduce:
add
workers = 8on bottom of /etc/odoo/odoo.conf
Current behavior:
after doing above operation, it get lots of error (as images shown)
Expected behavior:
working normally
Video/Screenshot link (optional):
login
after login
The text was updated successfully, but these errors were encountered: