Trying to write octets to a flexi-stream fails.

[informatimago]

The stream is created by drakma:http-request with :force-binary t We can seen in the inspect below that it is "wellformed" AFAIK, having ELEMENT-TYPE: FLEXI-STREAMS:OCTET. But when my code sends it bytes with write-sequence, it rejects by octets, expecting characters:

debugger invoked on a TYPE-ERROR @53CD9B82 in thread #<THREAD "main thread" RUNNING {1001440113}>: The value 92 is not of type CHARACTER

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Exit debugger, returning to top level.

((:METHOD FLEXI-STREAMS::WRITE-SEQUENCE* (FLEXI-STREAMS::FLEXI-LATIN-1-FORMAT T T T T)) #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument>) [fast-method]
   source: (CHAR-CODE CHAR-GETTER)
0] backtrace

Backtrace for: #<SB-THREAD:THREAD "main thread" RUNNING {1001440113}>
0: ((:METHOD FLEXI-STREAMS::WRITE-SEQUENCE* (FLEXI-STREAMS::FLEXI-LATIN-1-FORMAT T T T T)) #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument>) [fast-method]
1: ((:METHOD TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE (FLEXI-STREAMS:FLEXI-OUTPUT-STREAM T T T)) #<unavailable argument> #(92 59 220 141) #<unavailable argument> #<unavailable argument>) [fast-method]
2: ((SB-PCL::EMF TRIVIAL-GRAY-STREAMS:STREAM-WRITE-SEQUENCE) #<unused argument> #<unused argument> #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> #(92 59 220 141) 0 4)
3: ((:METHOD SB-GRAY:STREAM-WRITE-SEQUENCE (TRIVIAL-GRAY-STREAMS:FUNDAMENTAL-OUTPUT-STREAM T)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> #(92 59 220 141) 0 NIL) [fast-method]
4: (SB-IMPL::WRITE-SEQ-IMPL #(92 59 220 141) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> 0 NIL)
5: (WRITE-SEQUENCE #(92 59 220 141) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> :START 0 :END NIL)
6: ((:METHOD SEND-OCTETS (STREAM T)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> #(92 59 220 141) :START 0 :END NIL) [fast-method]
7: ((SB-PCL::EMF SEND-OCTETS) #<unused argument> #<unused argument> #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> #(92 59 220 141))
8: ((:METHOD CL-NAIVE-WEBSOCKETS::WRITE-FRAME (T CL-NAIVE-WEBSOCKETS::FRAME)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> #<CL-NAIVE-WEBSOCKETS::FRAME {1006695E43}>) [fast-method]
9: ((:METHOD CL-NAIVE-WEBSOCKETS::SEND-FRAME (WEBSOCKETS-ENDPOINT CL-NAIVE-WEBSOCKETS::FRAME)) #<WEBSOCKETS-CLIENT-ENDPOINT {10058355F3}> #<CL-NAIVE-WEBSOCKETS::FRAME {1006695E43}>) [fast-method]
10: ((FLET SB-THREAD::WITH-MUTEX-THUNK :IN CL-NAIVE-WEBSOCKETS::SEND-MESSAGE))
11: ((FLET "WITHOUT-INTERRUPTS-BODY-1" :IN SB-THREAD::CALL-WITH-MUTEX))
12: (SB-THREAD::CALL-WITH-MUTEX #<FUNCTION (FLET SB-THREAD::WITH-MUTEX-THUNK :IN CL-NAIVE-WEBSOCKETS::SEND-MESSAGE) {235721B}> #<SB-THREAD:MUTEX "websockets-endpoint %send-message-lock" owner: #<SB-THREAD:THREAD "main thread" RUNNING {1001440113}>> T NIL)
13: ((:METHOD CL-NAIVE-WEBSOCKETS::SEND-MESSAGE (WEBSOCKETS-ENDPOINT T T T)) #<WEBSOCKETS-CLIENT-ENDPOINT {10058355F3}> 1 #() NIL) [fast-method]
14: ((:METHOD SEND-TEXT-MESSAGE (WEBSOCKETS-ENDPOINT T)) #<WEBSOCKETS-CLIENT-ENDPOINT {10058355F3}> "" :MAY-FRAGMENT NIL) [fast-method]
15: ((SB-PCL::EMF SEND-TEXT-MESSAGE) #<unused argument> #<unused argument> #<WEBSOCKETS-CLIENT-ENDPOINT {10058355F3}> "")
16: ((FLET "G0" :IN CHAT))
17: (CHAT #<WEBSOCKETS-CLIENT-ENDPOINT {10058355F3}>)
18: (WS-CHAT-CLIENT "localhost" 1236 "/ws" :SECURE NIL :SUBPROTOCOL "chat" :ORIGIN NIL)
19: (RUN-CLIENT "localhost" 1236 "/ws" NIL "chat" NIL)
...



8: ((:METHOD CL-NAIVE-WEBSOCKETS::WRITE-FRAME (T CL-NAIVE-WEBSOCKETS::FRAME)) #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}> #<CL-NAIVE-WEBSOCKETS::FRAME {1006695E43}>) [fast-method]
8] list-locals
CL-NAIVE-WEBSOCKETS::DATA  =  #()
CL-NAIVE-WEBSOCKETS::FIRST-BYTE  =  129
CL-NAIVE-WEBSOCKETS::FRAME  =  #<CL-NAIVE-WEBSOCKETS::FRAME {1006695E43}>
CL-NAIVE-WEBSOCKETS::LEN  =  0
CL-NAIVE-WEBSOCKETS::LOWER-LAYER  =  #<FLEXI-STREAMS:FLEXI-IO-STREAM {1001FAFC83}>
CL-NAIVE-WEBSOCKETS::MASKING-KEY  =  #(92 59 220 141)
CL-NAIVE-WEBSOCKETS::PAYLOAD-LENGTH  =  0
CL-NAIVE-WEBSOCKETS::SECOND-BYTE  =  128

8] (inspect CL-NAIVE-WEBSOCKETS::LOWER-LAYER)

The object is a STANDARD-OBJECT of type FLEXI-STREAMS:FLEXI-IO-STREAM.
0. OPEN-P: T
1. STREAM: #<CHUNGA:CHUNKED-IO-STREAM {1001FACCD3}>
2. EXTERNAL-FORMAT: #<FLEXI-STREAMS::FLEXI-LATIN-1-FORMAT (:ISO-8859-1 :EOL-STYLE :LF) {100739F7C3}>
3. ELEMENT-TYPE: FLEXI-STREAMS:OCTET
4. COLUMN: NIL
5. LAST-CHAR-CODE: NIL
6. LAST-OCTET: NIL
7. OCTET-STACK: NIL
8. POSITION: 415
9. BOUND: NIL
> q

8] (write-sequence #(92 59 220 141)  CL-NAIVE-WEBSOCKETS::LOWER-LAYER)


debugger invoked on a TYPE-ERROR @53CD9B82 in thread #<THREAD "main thread" RUNNING {1001440113}>: The value 92 is not of type CHARACTER

Type HELP for debugger help, or (SB-EXT:EXIT) to exit from SBCL.

restarts (invokable by number or by possibly-abbreviated name):
  0: [ABORT] Reduce debugger level (to debug level 1).
  1:         Exit debugger, returning to top level.

((:METHOD FLEXI-STREAMS::WRITE-SEQUENCE* (FLEXI-STREAMS::FLEXI-LATIN-1-FORMAT T T T T)) #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument> #<unavailable argument>) [fast-method]
   source: (CHAR-CODE CHAR-GETTER)
0[2] 

[avodonosov]

The implementation seems to partially support binary output.
For example here:

flexi-streams/output.lisp

Line 147 in 74a1027

(when (and (vectorp sequence)

It seems, if the sequence you try to write was an array with integer element type, the output would work.

Also the docstring of that function:

An optimized version which uses a buffer underneath. The function
can accepts characters as well as octets and it decides what to do
based on the element type of the sequence (if possible) or on the
individual elements, i.e. you can mix characters and octets in
SEQUENCE if you want. Whether that really works might also depend on
your Lisp, some of the implementations are more picky than others."

Speaking of mixing the chars and octets in the sequence, the code seems to only support it for lists:

flexi-streams/encode.lisp

Line 129 in 74a1027

(list (iterate (write-object (nth index sequence))))))

For other input sequences, the code applies to every element a @body of code supplied as a parameter. See two lines above.

The body is expected to get the current element using char-getter symbol macro. And the actual bodies unconditionally pass the element to (char-code ):

flexi-streams/encode.lisp

Line 187 in 74a1027

(let ((octet (char-code char-getter)))

[informatimago]

Ok. Thank you for the answer.
Inded, when passing a (vector octet) it works, (vector T) are expected to contain only characters.

I find it regretable that libraries impose such restrictions, given that CL itself doesn't.

(with-open-file (out "/tmp/binary.out" :direction :output :element-type '(unsigned-byte 8) :if-does-not-exist :create :if-exists :supersede)
  (write-sequence #(65 66 67 68) out)
  (let ((buffer (make-array 4 :element-type '(or null string integer)
                              :initial-contents '(69 70 71 72))))
    (write-sequence buffer out)))
;; --> #(69 70 71 72)

(with-open-file (input "/tmp/binary.out" :element-type 'character)
  (read-line input))
;; --> "ABCDEFGH"
;;     t

so you may want to keep the issue to allow vectors of any type of element, as long as the contents is acceptable, or reject it (and close it).

For now I have a workaround, that makes it more costly to use flexi-stream: I just have to copy the data to temporary vector of octet buffers.

Thank you.

[avodonosov]

If anything, I am not a maintainer, just a subscribed user. Contributed some fixes in the past, so curious to look at issues sometimes.

[avodonosov]

So, in short, the issue is: flexi-streams, for input arrays whose element-type is not explicitly binary, always assumes characters. You want it instead to look at the stream element type of the flexi stream itself. Right?

That may be reasonable, especially that CLHS entry for write-sequence mentions stream element type in a way that may be interpretted to suggest such an aporoach:

Might signal an error of type type-error if an element of the bounded sequence is not a member of the stream element type of the stream.

But changing flexi-streams this way will break compatibility: somebody who was successfully performing character output on flexi-stream with element type octet will suddenly get a failure.

The most flexible would be to look at the actual type of every element, as it is currently implemented for lists, but I suppose that was avoided for arrays in a quest for higher performance.