[bug?] "Did not find CR at end of boundary"

[fosskers]

Thank you for reviving this package.

While it seems possible to initiate a POST with multipart/form-data, I'm hitting the dreaded problem where the server is cryptically reporting:

Did not find CR at end of boundary

when I submit the POST. I've searched around the net for people suffering the same symptoms in other contexts, but I haven't found a solution. My code looks like:

POST http://localhost:8000/runs
Content-Type: multipart/form-data; boundary=foobar123

--foobar123
Content-Disposition: form-data; name="foo"; filename="foo.txt"
Content-Type: application/octet-stream

< REDACTED/foo.txt

--foobar123
Content-Disposition: form-data; name="bar"; filename="bar.blob"
Content-Type: application/octet-stream

< REDACTED/bar.blob

--foobar123--

Does anything here stand out to you as strange? I'm following both your example under examples/ and also something similar from a coworker. Thank you.

[pederkl]

The error message seems fairly clear to me. Apparently your server insists on the canonical \r\n line endings, at least for boundaries. And it's not getting the \r (CR).

Restclient as it is today will parse the top-level headers (after POST) and hand them to the URL library separately, ensuring they are sent with the proper line endings, but has no facility for parsing the multipart headers.

Try changing your code to this:

POST http://localhost:8000/runs
Content-Type: multipart/form-data; boundary=foobar123

--foobar123^M
Content-Disposition: form-data; name="foo"; filename="foo.txt"^M
Content-Type: application/octet-stream^M
^M
< REDACTED/foo.txt

--foobar123^M
Content-Disposition: form-data; name="bar"; filename="bar.blob"^M
Content-Type: application/octet-stream^M
^M
< REDACTED/bar.blob

--foobar123--^M

Where each ^M is not the literal text, but the single character you get from typing C-q C-m in Emacs. Here, all the lines that are part of the protocol has the protocol-specified line endings.

If this solves your problem, I am considering implementing a custom variable to toggle forcing \r\n on each line in the body (file contents exluded).

[pederkl]

Oh, and I found a bug in the function that inserts the file content, that may bite you if your file contains backslashes. I just pushed a fix, but if you're too impatient to wait for melpa to update, please eval this after visiting your http file. (By "eval" I mean stick it in your scratch buffer, put your cursor in the middle of it and type C-M-x)

(defun restclient-replace-path-with-contents (entity)
  "Include file contents in request.
ENTITY is the request body, Possibly with `< /file/path' embeded, which
will be replaced with the contents of `/file/path'."
  (replace-regexp-in-string
   restclient-file-regexp
   (lambda (match)
     (restclient-read-file (match-string 1 match)))
   entity t t))

[fosskers]

I've pulled the latest from master and inserted the control characters, but still see the same error from the server.

[pederkl]

Hm. No more good ideas come to mind. And I can't reproduce the error using httpbin.org. What is the server you are connecting to? Is that something I can easily spin up locally to try to reproduce the error?

One thing to try is to (setq url-debug t), run your query, and then check the *URL-DEBUG* buffer. If you could save it to a text file and upload it (after redacting any non-public information, naturally), that might help by showing what the url library sees, control characters and all.

What version of Emacs are you running, and on what OS? I don't think url.el behaves differently, but who knows.

[fosskers]

This is the server https://pypi.org/project/fastapi/ but unfortunately the current work is proprietary so I can't share it. I'll keep trying to debug on my end for now.

[fosskers]

In *URL-DEBUG* I see:

http -> Contacting host: localhost:8000
http -> Marking connection as busy: localhost:8000 #<process localhost>
http -> getting referer from buffer: buffer:#<buffer  *temp*-913199> target-url:#s(url "http" nil nil "localhost" 8000 "/runs" nil nil t nil t t) lastloc:nil
http -> Request is: 
POST /runs HTTP/1.1^M
MIME-Version: 1.0^M
Connection: keep-alive^M
Host: localhost:8000^M
Accept-charset: <TRUNCATED by me>^M
Accept: */*^M
Content-Type: multipart/form-data; boundary=foobar123^M
Content-length: 595179^M
^M
--foobar123
Content-Disposition: form-data; name="foo"; filename="foo.txt"
Content-Type: application/octet-stream
...

Notice that the initial bits starting from the POST line do have ^M, but my lines starting from the first --foobar123 do not, even though I have them added manually in my org-mode src block. Yes, it should be mentioned that I'm running this through ob-restclient. Further, none of the lines of the injected text file foo.txt have ^M. Would that break it?

With the fixes you and the ob-restclient guy provided yesterday, I now also see a JSON response from the server:

{
  "detail": "There was an error parsing the body"
}

[pederkl]

That's interesting. POST and the following headers are treated specially by url.el, so it's no surprise that they have the ^M. And the contents of your files should not matter, that's just bytes.

But something stripping the ^Ms from your body shouldn't happen, it would be good to pinpoint what does that. It doesn't happen for me in my Emacs master build or with the 26.1 I use for testing. Can you try your query in a plain buffer in restclient-mode, to see if it's some interaction with org that does it?

[pederkl]

Can you please test the issues/4 branch I just pushed? It gives you a new option restclient-body-lines-end-in-crlf which you can set to t to force CRLF on body lines (except file contents).

I'm in two minds about installing this (assuming it fixes your problem). On the one hand, this doesn't seem to me to be a bug in restclient. Just pushing the buffer contents works for me. So ideally the root cause should be fixed instead of glossing over it with workarounds and hacks. And this is a hack. The body data could be anything, and restclient should not mess with what users try to send.

On the other hand, if it fixes your problem, that is one less problem. And if you need that line ending in your data, it is presumably nice to get it without those ugly ^Ms cluttering up your http files (or blocks).

Thoughts?

[fosskers]

Directly within a restclient buffer and not through ob-restclient, running restclient-http-send-current succeeds in the request (no more CRLF warning), but fails later, deeper in my server, for potentially unrelated reasons. If I remove all the ^M, I do still get the original error. If I remove ^M that I had added immediately after the < ... lines, I get a 422, which is apparently "Unprocessable Content".

So, ob-restclient may be trimming line endings before executing its content.

[pederkl]

Ok. So that sounds like an issue to raise with the ob-restclient project. But did you have an opportunity to test the branch I posted? With that, you should be able to get past the CRLF error even from your org buffer (after setting restclient-body-lines-end-in-crlf to t)

[fosskers]

I will test that on Tuesday when I'm back in the office.