binary http uploads are corrupted by p:http-request

[Conal-Tuohy]

#241 has the details

[ndw]

Sorry. I feel like I've lost the focus on this change and I did rework http-request to support Apache HTTP Client 2.5.x (because it was needed for https: to sites secured with Let's Encrypt certificates).

Can you provide a test that demonstrates the corruption?

[fgeorges]

Quick note, I guess you mean 4.5.* (not 2.5.*)...

[ndw]

Oops. Yes.

[Conal-Tuohy]

OK I found an "echo" server that will do the job. This test uploads a three-byte sequence (181, 181, 181) to an echo server and reports the bytes which the server says it received.

<p:declare-step version="1.0" 
   xmlns:p="http://www.w3.org/ns/xproc" 
   xmlns:c="http://www.w3.org/ns/xproc-step" 
   xmlns:j="http://marklogic.com/json" 
   name="http-binary-upload-test">
   <p:output port="result"/>
   <p:documentation>
      This test posts three bytes to an echo server on the internet.
      The bytes sent have identical values (181 = 0xB5).
      The byte sequence 0xB5 0xB5 0xB5 encoded in base64 = "tbW1".
   </p:documentation>
   <p:http-request name="post-three-bytes">
      <p:input port="source">
         <p:inline>
            <c:request href="http://echo.getpostman.com/post" method="POST">
               <c:body content-type="application/octet-stream" encoding="base64">
                  tbW1
               </c:body>
            </c:request>
         </p:inline>
      </p:input>
   </p:http-request>
   <p:unescape-markup name="parse-json-response">
      <p:with-option name="content-type" select="/c:body/@content-type"/>
      <p:with-option name="encoding" select="/c:body/@encoding"/>
   </p:unescape-markup>
   <p:template name="list-content-bytes-echoed-from-server">
      <p:with-param name="bytes" 
         select="string-join(/c:body/j:json/j:data/j:data/j:item, ', ')"/>
      <p:input port="template">
         <p:inline exclude-inline-prefixes="c j">
            <bytes>{$bytes}</bytes>
         </p:inline>
      </p:input>
   </p:template>
</p:declare-step>

My fork returns the expected result:

<bytes>181, 181, 181</bytes>

Calabash 1.1.14-97 returns:

<bytes>239, 191, 189, 239, 191, 189, 239, 191, 189</bytes>

[Conal-Tuohy]

In case you're wondering "why 181?", it was simply the first corrupted byte I discovered when I uploaded a PDF file and sniffed the network traffic.

[Conal-Tuohy]

A build of my fork is available here, by the way: https://github.com/Conal-Tuohy/xmlcalabash1/releases/tag/1.1.11-ctuohy