Request logging encoding stopped working in 2.0.x

[quantoid]

I'm using the --log-req-encoder option to encode request logging as JSON. This has been working fine for a while, but recently stopped having any effect (the logging is now plain text).

Options used:

fmt=$'{"timestamp": "${strftime:%FT%TZ}", "level": "DEBUG", "name": "uwsgi", "message": "${msg}"}\n'
uwsgi --need-app --module=myapp.main:app \
  --http=0.0.0.0:8080 --uid=leaf --master --workers=8 --enable-threads \
  --env="TZ=UTC" --log-req-encoder="json ${fmt}" --logformat="%(status) [%(msecs)ms] %(method) %(uri)"

What I see on stdout:

[log-req-encoder] registered json {"timestamp": "${strftime:%FT%TZ}", "level": "DEBUG", "name": "uwsgi", "message": "${msg}"}
:::
spawned uWSGI http 1 (pid: 76846)
{"timestamp": "2017-10-02T23:11:54.135Z", "message": "Descriptor requested", "request": "1470df66-a7c7-11e7-b483-186590cba8eb", "name": "myapp.main", "level": "INFO"}
200 [11ms] GET /descriptor.json

The JSON encoded line is the output from my Flask app, which uses a python log formatter to encode as JSON.

The next line logging the GET request/response used to be JSON encoded but is not any more.

[quantoid]

I'd also be interested to know how to use --log-encoder to encode stdout as JSON without also encoding the output from the Flask app (which is already JSON encoded).

[jwineinger]

@quantoid did you have any luck with the second part (not encoding your flask app output)? I also found that I had to go back to 2.0.14 to get any of the json encoding to work

[f-f]

+1 on this, I'm running into the same issues: I'm defining json logger throughout all of our stack, and I cannot make it work on uwsgi.

I would say there is two different issues here:

1. It is not possible to define log-req-encoder without having to define also a log-encoder
2. It is not possible to "passthrough" the app logs

[quantoid]

Finally solved this today: you have to include —logger-req=stdio and then the json encoder starts working.

[quantoid]

But still haven’t found a way to stop app logging from being encoded along with other stdout.

[quantoid]

What I think we need here is an option to redirect stdout and stderr from uwsgi itself (but not from the app) to a logger, with the appropriate logging level. Then you can keep control of where all your logging goes and how it gets formatted in the usual python way.

The output from the app (and hence from its loggers) should be left alone in this case, because it's probably already output from loggers. I guess I could configure the root logger to just output the message to stdout and rely on uwsgi formatting as logging, but then I lose the name and level attributes.

[jwineinger]

I agree; I also found it very difficult to get JSON logs out of uwsgi and my application together without having the application logs encoded to a string. I ended up writing a plugin that defines new logchunks ( http://uwsgi-docs.readthedocs.io/en/latest/LogFormat.html#hacking-logformat) which JSON encode the request variables that I wanted to output in my logs. Then I defined my log-format carefully to be a JSON object and used those new variables. An excerpt: log-format = {"src":"uwsgi","msg":{"uri":"%(json_uri)","host":"%(json_host)","uagent":"%(json_user_agent)" ... }} This makes uwsgi's log messages appear as JSON. Then I configured my python app to output its logs as JSON, which appear on separate lines from the uwsgi log messages.

…

On Tue, Feb 27, 2018 at 9:04 PM, Mike ***@***.***> wrote: What I think we need here is for stdout and stderr from uwsgi itself (but not from the app) to be sent to a logger with the appropriate logging level. Then you can keep control of where all your logging goes and how it gets formatted in the usual python way. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1654 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AARhuf81gf6BdYBqXZNesTCO03queexMks5tZMJbgaJpZM4PrYyt> .

[iborko]

I've recently managed to configure uWSGI to output all logs in JSON format and pass-through the application logs (as they are already in JSON format). I also ended up writing a uWSGI logging plugin with custom logchunks. You can read more about it on my blog: https://www.velebit.ai/blog/tech-blog-uwsgi-json-logging/

There is also a link on my repo that contains the source code of my uWSGI plugin and all required instructions to build and use it.