Should charCodeAt be used for binary IO?

[ejgallego]

charCodeAt is pervasively used in the javascript library to read a byte from a string.

However, such use is may be inconsistent when strings are used in OCAML to contain binary data, as charCodeAt may return values higher than 256. For instance:

> var str = "ℕlala"
> var utf8 = unescape(encodeURIComponent(str));
> var arr = [];
> for (var i = 0; i < utf8.length; i++) { arr.push(utf8.charCodeAt(i)); };
> arr
[ 226, 132, 149, 108, 97, 108, 97 ]
> utf8
'�lala'
> str.charCodeAt(1);
108
> utf8.charCodeAt(1);
132
> arr[1];
132

This may break some binary reading done in Ocaml, like in dynlink.ml, etc...

After a bit of thinking about Ocaml strings, it is not clear to me what should be the best choice here. One option is to implement strings always as byte arrays in JS, replacing charCodeAt by regular array access.

[hhugo]

pervasively used when the string is correctly encoded (sequence of byte).
I don't know how you get your strings, but I suspect they are sent to the client not been utf8 encoded.

see string representation:
https://github.com/ocsigen/js_of_ocaml/blob/master/runtime/mlString.js#L25

and the utf8 encoding
https://github.com/ocsigen/js_of_ocaml/blob/master/lib/deriving_json/deriving_Json.ml#L198

[hhugo]

I'll push an example of dynlink in the toplevel later today.

[vouillon]

If you are getting your data through XmlHttpRequest, you can ask for the response to be stored in an ArrayBuffer, and then copy the buffer contents into an OCaml string with the following function:

let string_of_buffer b =
  let a = jsnew Typed_array.uint8Array_fromBuffer(b) in
  let len = a##length in
  let s = String.create len in
  for i = 0 to len - 1 do
    s.[i] <- Char.chr (Typed_array.unsafe_get a i)
  done;
  s

(See https://developer.mozilla.org/en-US/docs/Web/API/XMLHttpRequest/Sending_and_Receiving_Binary_Data )

In particular, the function load_resource in toplevel/toplevel.cppo.ml only works with ASCII data.

[ejgallego]

Thank you for the explanation and help.

I finally managed to load binary data with a synchronous XHR request, but it was a bit of pain.

I was wondering thou if requiring the ocaml side to use -safe-string would make things a bit better in the js_of_ocaml side, as then Ocaml strings could be mapped to js strings and ocaml Bytes to js ArrayBuffers.

Best,
E.

[zoggy]

Hi, FYI you might be interested in this PR in cohttp and give it a try. (if you use cohttp of course).