Opencast Eats + flavour characters in some cases

[gregorydlogan]

Describe the bug
Ingesting to Opencast should be something you can do more or less in any order:

• createMediaPackage
• addTrack
• addCatalog
• addTrack
• ...
• ingest

This does work, as long as you don't have a flavour containing a + character. In that case the file(s) containing the + must go last, or else the + characters are replaced with spaces in the flavours. Media files don't seem to affect this, which makes me suspect it's something in the catalog ingest handling code

I noted this behaviour during the Zoom ingest development, and Lukas Germscheid reported it on list as well.

To Reproduce
Steps to reproduce the behavior:

1. Create a mediapackage
2. Add a catalog with flavour security/xacml+episode
3. Add a catalog with flavour dublincore/episode
4. Check the resulting mediapackage

Looking at the commit (not public yet), my notes say:

    Fixing incorrect xacml flavour issue.
    
    This appears to be related to the order in which ingests are done.  If you
    ingest files such that a call to _do_post occurs *without* files containing
    something *after* you ingest a file containing a '+' character in its flavour
    you will see it magically disappear the + from the next post.  The major issue
    debugging this is that *this only disappears _after_ the post is sent*.  It is
    *not* visible from the Python side, but it shows up in the debugger in Java.
    
    I suspect this is the requests_toolbelt setting an incorrect content type,
    since '+' renders as ' ' depending on the content type.  Further debugging required.

which makes me suspect that it's something related to content types and in the catalog ingest bits.

Environment Information

• OS: [e.g. iOS 8.1, CentOS 7]: All
• Browser [e.g. Safari, Chrome]: n/a
• Opencast version with minor release [e.g. 7.0, 7.2, 7.x if using a branch]: 9.x, 10.x

[Rillke]

Link back to Google Groups discussion.

[mtneug]

This is an escaping problem. Some endpoints directly handle the request object some let the Java libraries do it. In the latter cases, you need to URL encode all form fields even when using multipart/form-data. We even have one of the helper scripts doing this. With multipart/form-data this should normally not be necessary.

[gregorydlogan]

This appears to be happening in the returned data from Opencast, rather than the incoming requests. It's likely still a escaping issue though.

[mtneug]

Well if the incoming request is parsed incorrectly the returned data is wrong. You can debug this: e.g. set a break point here and you will see that mp strips the + from the request when not URL encoded.