Test failure with SBCL 2.4.2

[MIvanchev]

I run the tests as follows:

       sbcl --non-interactive \
            --eval '(require "asdf")' \
            --eval "(push #p\"${wrksrc}/\" asdf:*central-registry*)" \
            --eval '(asdf:load-system "trivial-gray-streams")' \
            --eval '(asdf:load-system "trivial-gray-streams-test")' \
	    --eval '(in-package :trivial-gray-streams-test)' \
            --eval "(when (failed-test-names (run-tests)) (uiop:quit 1))"

and the output is

This is SBCL 2.4.2.void.1, an implementation of ANSI Common Lisp.
More information about SBCL is available at <http://www.sbcl.org/>.

SBCL is free software, provided as is, with absolutely no warranty.
It is mostly in the public domain; some portions are provided under
BSD-style licenses.  See the CREDITS and COPYING files in the
distribution for more information.
Running test STREAM-READ-CHAR... OK
Running test STREAM-UNREAD-CHAR... OK
Running test STREAM-READ-CHAR-NO-HANG... OK
Running test STREAM-PEEK-CHAR... OK
Running test STREAM-LISTEN... OK
Running test STREAM-READ-LINE... OK
Running test STREAM-CLEAR-INPUT... OK
Running test STREAM-WRITE-CHAR... OK
Running test STREAM-LINE-COLUMN... OK
Running test STREAM-START-LINE-P... OK
Running test STREAM-WRITE-STRING... OK
Running test STREAM-TERPRI... OK
Running test STREAM-FRESH-LINE... OK
Running test STREAM-FINISH-OUTPUT... OK
Running test STREAM-FORCE-OUTPUT... OK
Running test STREAM-CLEAR-OUTPUT... OK
Running test STREAM-ADVANCE-TO-COLUMN... FAIL: expected invocation: (sb-gray:stream-advance-to-column
                                                                     #<trivial-gray-streams-test::test-stream2 {10016a6d93}>
                                                                     10) actual: (sb-gray:stream-line-column
                                                                                  #<trivial-gray-streams-test::test-stream2 {10016a6d93}>), (sb-gray:stream-write-string
                                                                                                                                             #<trivial-gray-streams-test::test-stream2 {10016a6d93}>
                                                                                                                                             "  "
                                                                                                                                             0
                                                                                                                                             2)
Running test STREAM-READ-BYTE... OK
Running test STREAM-WRITE-BYTE... OK
Running test STREAM-READ-SEQUENCE... OK
Running test STREAM-WRITE-SEQUENCE... OK
Running test STREAM-FILE-POSITION... OK
Running test SETF-STREAM-FILE-POSITION... OK

Could you help me figure out what's going wrong?

[avodonosov]

That's a known failure. Nothing is wrong.

The failing test case validates that the lisp implementation calls stream-advance-to-column when executing format ~t.

This expectation is based on the Gray proposal wording:

STREAM-ADVANCE-TO-COLUMN stream column [Generic Function]

[...] This is intended for use by by PPRINT and FORMAT ~T.

All lisp implementations fail this.

https://cl-test-grid.common-lisp.dev/library/trivial-gray-streams.html

PS. In your script the --eval '(asdf:load-system "trivial-gray-streams")' is redundant, as the trivial-gray-streams-test system has trivial-gray-streams as a dependency.

[MIvanchev]

Maybe the failure could be dealt with in the test's body then so the test passes as OK? I need to distinguish actually failing tests and I don't really want to make an exception for this one because it could be failing for some other reason.

[MIvanchev]

I also don't quite get it. Are these tests testing the CL implementation or this library?

[MIvanchev]

I've changed the code to

	sbcl --non-interactive \
	     --eval '(require "asdf")' \
	     --eval "(push #p\"${wrksrc}/\" asdf:*central-registry*)" \
	     --eval '(asdf:load-system "trivial-gray-streams-test")' \
	     --eval '(in-package :trivial-gray-streams-test)' \
	     --eval '(when (not (equal (list "stream-advance-to-column")
	                               (failed-test-names (run-tests))))
	               (uiop:quit 1))'

But it still feels a bit weird.

[avodonosov]

@MIvanchev

Maybe the failure could be dealt with in the test's body then so the test passes as OK?

Test suite can include this information. The test will not be passing OK, but it could be marked as a known failure (many test frameworks have a concept of known failures). For example it could be a (defvar *known-failures* '((:sbcl test-a, test-b) (:abcl test-x, test-y)...), or some annotation in the test definition like :fails-on (:sbcl, :ccl ...). It's just that so far I was the only user of the test results, and I didn't need that.

I can't promise I will add this very quickly. Especially that you already have a solution for your scripts.

I also don't quite get it. Are these tests testing the CL implementation or this library?

The tests validate that tirival-gray-streams works as a portability layer. That if someone defines a stream class that implements, say, stream-write-char, the application code invoking write-char results in the stream-write-char invoked. For that goal it's not reasonable to pursue strict modularity boundaries - this library or cl implementation - we test both together (although mostly CL implementation, because this library basically just re-exports symbols from an implementation specific package into the trivial-gray-streams package).

[MIvanchev]

So I "fixed" it in the template by checking that it's the only test that fails

sbcl --non-interactive \
	     --eval '(require "asdf")' \
	     --eval "(push #p\"${wrksrc}/\" asdf:*central-registry*)" \
	     --eval '(asdf:load-system "trivial-gray-streams-test")' \
	     --eval '(in-package :trivial-gray-streams-test)' \
	     --eval '(when (not (equal (list "stream-advance-to-column")
	                               (failed-test-names (run-tests))))
	               (uiop:quit 1))

It's still not a perfect solution because of said reason that the test might be failing for other reasons but it's a step forward :) I'm closing the issue.

For that goal it's not reasonable to pursue strict modularity boundaries - this library or cl implementation - we test both together (although mostly CL implementation, because this library basically just re-exports symbols from an implementation specific package into the trivial-gray-streams package).

Ah, OK, I was missing the re-export semantics. So basically to test trivial-gray-streams own code would require an implementation not supporting any gray streams?

[avodonosov]

So basically to test trivial-gray-streams own code would require an implementation not supporting any gray streams?

No, trivial-gray-streams only work for implementations with gray streams.

There is a little area where trival-gray-streams needs some code code. To support 3 functions that can not be just re-exported: stream-read-sequence, stream-write-sequence, file-position.

These functions are absent in the Gray proposal (I guess because the Common Lisp the Language book does not have cl:read-sequence, cl:write-sequence, cl:file-position), and in result Lisp implementations disagree on naming and parameters for these functions. For example old CLISP had gray:stream-read-byte-sequence, gray:stream-read-char-sequence, CCL has ccl:stream-read-vector, ccl:stream-read-list.

So trivial-gray-steams unifies them by defining its own generic functions stream-read-sequence, stream-write-sequence, file-position. Then creates methods for implementation specific functions (like ccl:stream-read-vector), which call the unified versions (like trivial-gray-streams:stream-read-sequence).

application -> cl:read-sequcence -> ccl:stream-read-vector -> trivial-gray-streams:stream-read-sequence.

For the implementation specific methods be applicable to all user-defined stream classes, based on trivial-gray-streams, we need to mirror all Gray class hierarchy - define our own classes inherited from implementation specific Gray classes, instead of simply re-exporting the implementation classes.

All this support for the 3 functions is implemented in the streams.lisp.