handling non-standard post data format with filtering

[MeredithAnya]

Sending post data in either JSON or urlencoded formats will correctly filter sensitive data through filter_http (code here).

However, something like a JSON serialized string representing the data gets mutated awkwardly:

{'data': '"{"meredith":{"first_name":"gosh","last_name":"darn"},"commit":"Save Meredith","authenticity_token":"HyhO0Ut+vb6QeZ2f1HadNu1MsNctx7HiA/q5mIWYqA81MctGBCw0ZYEIsI84K+USqcLdx6yHsdpCFViOYqlwPw="}"'}

becomes

{'data': '"{"meredith":{"first_name":"gosh","last_name":"darn"},"commit":"Save Meredith","authenticity_token":"HyhO0Ut+vb6QeZ2f1HadNu1MsNctx7HiA/q5mIWYqA81MctGBCw0ZYEIsI84K+USqcLdx6yHsdpCFViOYqlwPw=[Filtered]'}

Not only does this not filter the authenticity_token correctly, it also cuts off a trailing } which causes problems in the UI in rendering the post data.

note: this example fails because of the single =

cc @tkaemming

[tkaemming]

There are a few options on how to solve this (or at least make it less painful):

• Have the SDK send the data in the most analogous data type (in this case, a JSON mapping — but not serialized into string like it is here),
• Have the SDK send the body in it's original form (in this case, urlencoded) and decode it for scrubbing based on the value of the HTTP Content-Type header,
• Give up on the SDK providing the data in the correct format, and if a string value is provided for the data attribute, try and parse it as JSON before continuing with the existing behavior that attempts to urldecode it.

Identifying what is going on here was complicated by the fact that the frontend will try and render anything that looks like JSON as JSON.

[evanpurkhiser]

I think this is probably our best bet:

1. Accept anything in the data fields from clients. We should (and I'll describe that later) update our documentation to specify how clients should send JSON, URL data, and arbitrary data, and then have the clients updated to do this, but I think ted's last point is probably going to be the path that works "the best":

   Give up on the SDK providing the data in the correct format, and if a string value is provided for the data attribute, try and parse it as JSON before continuing with the existing behavior that attempts to urldecode it.

   • Here's the variations we'll have on data:

     • JSON already decoded and assigned to the data value as an object.
     • JSON encoded as a string on data.
     • URL encoded values decoded and assigned to the data value as an object.
     • URL encoded values as a string on data
     • Some other arbitrary data, we don't do anything to handle this.

   • If the data is a string we can inspect to see if it looks like JSON or if it looks like url encoded values. We can then use this to decode it and always pass the data along as an object to the sentry web app. In this case we'll have to add an extra field to the client API to specify what the detected type was so the app knows what to do with it (what does curl look like, see x-www-form-urlencoded mistaken for JSON #5673), as we may not be able to simply rely on the Content-Type header.

   • If the data is provided to us as an object from the raven client, we can't know just from the data if it was url encoded or a JSON object. In this case we must inspect the Content-Type header, and hope that it's correct, as we'll be using that as the body content type passed to the client.

2. The sentry frontend app should be updated to read the detected body type field and use that to decide how to render content. This means we'll be pushing off the logic Ted mentioned about 'trying to render anything that looks like JSON' up to the server and keeping it in one place.

3. Update our documentation on the request interface and our SDKs.

   In our documentation we specifically say anything can go in data. This should be updated to say something more along the lines of:

   Prefer the string representation of the body content. Objects may also be used, however, the content type must be correct for content such as JSON and URL encoded form data to be represented in the sentry UI correctly.

The SDKs can then be updated to do this. Raven Python already accidentally does this: getsentry/raven-python#1067.

/cc @tkaemming for thoughts.