Twitcher unable to handle large http POST?

[tlvu]

@dbyrns

First off, sorry to hijack Magpie for Twitcher question, I was not able to find the "Issues" tab on Ouranos twitcher repo (https://github.com/Ouranosinc/twitcher).

Image: pavics/twitcher:pavics-0.3.16.

In this gist https://gist.github.com/tlvu/a9f63c90b6538050d3e9d432b5d276f1 (https://nbviewer.jupyter.org/gist/tlvu/a9f63c90b6538050d3e9d432b5d276f1), there is a Raven notebook that submit a rather large/gigantic POST to Raven wps through twitcher DEBUG:urllib3.connectionpool:https://lvu.ouranos.ca:443 "POST /twitcher/ows/proxy/raven/wps HTTP/1.1" 413 199 and the resulting is XMLSyntaxError: Opening and ending tag mismatch: hr line 5 and body, line 6, column 8.

Bypassing twitcher and hitting the Raven wps directly works so it's twitcher that was unable to process the request.

With twitcher in front, tracing Raven, the request did not even made it to Raven. Tracing twitcher at Debug level, there is no trace of this large/gigantic POST coming in either.

Any suggestions, debugging tips? Is this a known issue? Would upgrading to latest and greatest twitcher fix this issue?

Edit:

Http code 413 is indeed "Payload Too Large", see https://httpstatuses.com/413

Relevant Server Logs:

==> /opt/birdhouse/var/log/supervisor/nginx.log <==
2019/06/11 18:32:42 [error] 72#72: *9 client intended to send too large body: 1158967 bytes, client: 172.25.0.1, server: localhost, request: "POST /ows/proxy/raven/wps HTTP/1.0", host: "lvu.ouranos.ca"

[fmigneault]

Try posting it there :
https://github.com/bird-house/twitcher/issues

[tlvu]

Try posting it there :
https://github.com/bird-house/twitcher/issues

Since we are using Ouranos twitcher and there are much differences between Ouranos twitcher and bird-house twitcher, I felt I had to ask you guys at the CRIM who maintain Ouranos twitcher.

But sure I can post on bird-house twitcher, maybe they have some useful clues.

[dbyrns]

In fact, the 1.1.0 version almost use the stock birdhouse twitcher. @fmigneault could confirm, but by config, we can even remove the magpie hook completely. However for the 0.3.16, I agree that the difference is not trivial and the issue you are experiencing could be unrelated to birdhouse. If you are working on upgrading twitcher to the latest version, you could check against that version before posting the issue directly to the birdhouse repo.

[tlvu]

Is it possible for the CRIM to share the docker-compose.yml and all config files used for your latest twitcher/magpie combo?

In fact ... are you guys source controlling your various docker-compose.yml somewhere, can I just have access to that repo?

Don't want to spend time tracking down which config have changed/added/deleted compared to our very ancient twitcher/magpie combo.

[fmigneault]

docker-compose should be something like:

  magpie:
    image: pavics/magpie:1.1.0
    container_name: magpie
    ports:
      - "2001:2001"
    environment:
      HOSTNAME: $HOST_FQDN
      FORWARDED_ALLOW_IPS: "*"
    depends_on:
      - postgres
    links:
      - postgres
    volumes:
      - ./config/magpie/permissions.cfg:/opt/local/src/magpie/config/permissions.cfg
      - ./config/magpie/providers.cfg:/opt/local/src/magpie/config/providers.cfg
      - ./config/magpie/magpie.env:/opt/local/src/magpie/magpie/env/magpie.env
      - ./config/postgres/credentials.env:/opt/local/src/magpie/magpie/env/postgres.env
    restart: always

  twitcher:
    image: pavics/twitcher:magpie-1.1.0
    container_name: twitcher
    ports:
      - "8000:8000"
    depends_on:
      - postgres
      - magpie
    links:
      - postgres
    env_file:
      - ./config/postgres/credentials.env
    volumes:
      - ./config/twitcher/twitcher.ini:/opt/birdhouse/src/twitcher/twitcher.ini
    command: 'pserve /opt/birdhouse/src/twitcher/twitcher.ini'
    restart: always

new file twitcher.ini.template, something like following:
(variables must be written with pavics-compose.sh)

###
# app configuration
# https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/environment.html
###

[app:main]
use = egg:pyramid_twitcher

pyramid.reload_templates = true
pyramid.debug_authorization = false
pyramid.debug_notfound = false
pyramid.debug_routematch = false
pyramid.default_locale_name = en
pyramid.includes =
sqlalchemy.url = postgresql://${POSTGIS_USER}:${POSTGIS_PASSWORD}@postgis:${POSTGIS_PORT}/twitcher

retry.attempts = 3

# By default, the toolbar only appears for clients from IP addresses
# '127.0.0.1' and '::1'.
# debugtoolbar.hosts = 127.0.0.1 ::1

# twitcher
twitcher.url = https://${HOST_FQDN}/twitcher
twitcher.adapter = magpie.adapter.MagpieAdapter
twitcher.rpcinterface = false
twitcher.username =
twitcher.password =
twitcher.ows_security = true
twitcher.ows_proxy = true
twitcher.ows_proxy_delegate = false
twitcher.ows_proxy_protected_path = /ows
twitcher.workdir =
twitcher.prefix =

# magpie (for twitcher.adapter)
magpie.url = https://${HOST_FQDN}/magpie
magpie.secret = ${MAGPIE_SECRET}
magpie.admin_user = ${MAGPIE_ADMIN_USER}
magpie.admin_password = ${MAGPIE_ADMIN_PW}

###
# wsgi server configuration
###

[alembic]
# path to migration scripts
script_location = twitcher/alembic
file_template = %%(year)d%%(month).2d%%(day).2d_%%(rev)s
# file_template = %%(rev)s_%%(slug)s

[server:main]
use = egg:waitress#main
listen = 0.0.0.0:8000

###
# logging configuration
# https://docs.pylonsproject.org/projects/pyramid/en/latest/narr/logging.html
###

[loggers]
keys = root, twitcher, sqlalchemy

[handlers]
keys = console

[formatters]
keys = generic

[logger_root]
level = INFO
handlers = console

[logger_twitcher]
level = INFO
handlers =
qualname = twitcher

[logger_sqlalchemy]
level = WARN
handlers =
qualname = sqlalchemy.engine
# "level = INFO" logs SQL queries.
# "level = DEBUG" logs SQL queries and results.
# "level = WARN" logs neither.  (Recommended for production systems.)

[handler_console]
class = StreamHandler
args = (sys.stderr,)
level = NOTSET
formatter = generic

[formatter_generic]
format = %(asctime)s %(levelname)-5.5s [%(name)s:%(lineno)s][%(threadName)s] %(message)s

Twitcher's port was changed.
It can be mapped to the old one or be updated in the nginx config files.

[tlvu]

Thanks @fmigneault what about config file(s) for magpie? We currently have pavics/magpie:0.7.3.

[tlvu]

Looking at pavics/twitcher images on Docker hub (https://hub.docker.com/r/pavics/twitcher/tags), tag magpie-1.1.0 has been built 14 days ago but in the https://github.com/Ouranosinc/twitcher repo, the last commit was in Decembre 2018 !

How did you guys build this magpie-1.1.0 exactly? Not using Docker Hub autobuild? Where is the code change committed?

[tlvu]

Got a hackish fix here https://github.com/Ouranosinc/PAVICS/tree/fix-twitcher-unable-to-handle-large-http-POST. The Raven on my PAVICS is now working properly behind the existing twitcher.

from birdy import WPSClient
url = "https://lvu.ouranos.ca/twitcher/ows/proxy/raven/wps"
wps = WPSClient(url, verify=False)  # I use self-signed SSL

The fix was basically to set a bigger client_max_body_size 20M for nginx, default was 1M only.

I suspect the same fix will be needed for newer twitcher. @fmigneault are you able to bundle that fix in your newer twitcher?

Still waiting for the magpie config files for newer magpie. I don't want to waste time chasing config changes like in #172 (comment).

[fmigneault]

@tlvu
Magpie uses the same setup with magpie.env file mounted as shown in the compose.
The permissions.cfg file can be omitted and providers.cfg remains as is.

Mapgie's docker now automatically generates the corresponding Twitcher image with Magpie adapter already setup in it, and this on every master merge / tag push.
It uses dockerhub autobuild hook: https://github.com/Ouranosinc/Magpie/tree/master/hooks

[fmigneault]

@tlvu
I think the fix only applies on PAVICS side now.
The new Twitcher doesn't use nginx at all, just a waitress process running the pyramid app.

[dbyrns]

I would add that the main nginx and the only one when using the latest twitcher is already accepting huge request body : https://github.com/Ouranosinc/PAVICS/blob/master/birdhouse/config/proxy/nginx.conf#L30

[tlvu]

Mapgie's docker now automatically generates the corresponding Twitcher image with Magpie adapter already setup in it, and this on every master merge / tag push.
It uses dockerhub autobuild hook: https://github.com/Ouranosinc/Magpie/tree/master/hooks

Oh right, I remember now we discussed about this. Sorry, bad memory.

Can you then maybe reference the https://github.com/Ouranosinc/Magpie repo on the Docker Hub of pavics/twitcher image https://hub.docker.com/r/pavics/twitcher to avoid confusion? Right now it references https://www.github.com/Ouranosinc/twitcher and it's very easy to confuse user (including myself).

[fmigneault]

@tlvu
The github link points to birdhouse's Twitcher because it uses that one, but the images in the https://github.com/Ouranosinc/Magpie#docker-images section uses the pavics/twitcher tags.

[tlvu]

I would add that the main nginx and the only one when using the latest twitcher is already accepting huge request body : https://github.com/Ouranosinc/PAVICS/blob/master/birdhouse/config/proxy/nginx.conf#L30

@dbyrns: FYI @moulab88 suggest not to have a too large client_max_body_size to prevent "Buffer overflow attacks" vulnerability.

[dbyrns]

I would add that the main nginx and the only one when using the latest twitcher is already accepting huge request body : https://github.com/Ouranosinc/PAVICS/blob/master/birdhouse/config/proxy/nginx.conf#L30

@dbyrns: FYI @moulab88 suggest not to have a too large client_max_body_size to prevent "Buffer overflow attacks" vulnerability.

Yep, when I saw that comment I immediately thought to the main nginx config . We could lower it as needed as well. I don't have in mind a specific scenario requiring such a high value, probably set that high just to get rid of errors for good.

[tlvu]

The github link points to birdhouse's Twitcher because it uses that one, but the images in the https://github.com/Ouranosinc/Magpie#docker-images section uses the pavics/twitcher tags.

@fmigneault: I was referring to the github link on Ouranos twitcher docker image pavics/twitcher. That github link should probably point to Magpie repo now since the matching code is there.

[fmigneault]

If you mean the Recent builds section here:
https://cloud.docker.com/u/pavics/repository/docker/pavics/twitcher/general
I can't change it.

Note that autobuilds from https://github.com/Ouranosinc/twitcher are not disabled, tags pushed to this repo will still build corresponding images.

[tlvu]

Note that autobuilds from https://github.com/Ouranosinc/twitcher are not disabled, tags pushed to this repo will still build corresponding images.

Oh ok. Not sure why we would still want that given we prefer to stick with upstream birdhouse twitcher as much as we can.

I guess once we can remove this autobuild from https://github.com/Ouranosinc/twitcher, we can switch this autobuild link to https://github.com/Ouranosinc/Magpie (new tags we publish there are magpie-#.#.# tags only).

[dbyrns]

Yeah things change a lot since then. I propose that @fmigneault looks at the PAVICS compose and starts a branch with a basic config working with the latest 1.3.x magpie image. You could then start from there and complete the integration with PAVICS. Indeed we will still be available to help you in the process.

[tlvu]

Thanks @dbyrns