Compile/build problem on RasPi3

[KenDickey]

Greetings,

First, thanks much for making Chez Scheme open source!!

I have been trying to get Chez up on a Raspberry Pi 3 under Raspian Linux, roughly following the guide at:

https://programmingpraxis.com/2017/09/15/compile-chez-scheme-on-android-arm/

The basic REPL is fine for small things, but some library loads fail with

  Exception in +: #f is not a number

The test suite also shows this behavior.

Perhaps a codegen in FFI call?

Any suggestions?
Thanks much,
-KenD

akku-compile-bug.txt
mats-summary.txt

[akeep]

It is very difficult to tell what is going on, except to say that the compiler is likely generating some bad code or the garbage collector is getting some corrupted data from the generated code which is causing it to overwrite something.

One thing to try is to compile the compiler and built-in libraries at optimize level 2:

$ make o=2 allx

If there is something wrong with the compiler backend for this target, it is likely you'll see some exceptions pop up during the compilation of the compiler and library code.

Debugging this kind of thing usually means trying to minimize to as small an example as possible and then running Chez Scheme under gdb (or lldb on macOS, though I've not used lldb much).

I do not have a Raspberry Pi 3 to test on, and currently the only ARM-based machine I have is not setup for this kind of development, but if I get some time I'll see if I can get an emulator running that I can try to replicate this in. The failing mats are pretty indicative that things are going wrong.

I'm assuming this is the 32-bit ARM target that you are using on the Raspberry Pi 3?

[KenDickey]

On 2019-06-25 20:16, Andy Keep wrote: ...

One thing to try is to compile the compiler and built-in libraries at optimize level 2: $ make o=2 allx If there is something wrong with the compiler backend for this target, it is likely you'll see some exceptions pop up during the compilation of the compiler and library code.

Great. Thanks much. I'll hook up my RasPi3 and check it out.

I'm assuming this is the 32-bit ARM target that you are using on the Raspberry Pi 3?

Yes. Raspian Linux 32 bit. I don't use it much. My daily use machines are arm64 (LePotato and Chromebook Plus). The arm7/32bit is basically to get confidence and learn process/tools, but I have to make things visible to develop. A bit of debug to get Chez up on arm32/RasPi would not be bad for the community. 8^) Thanks for the help! -KenD

[weinholt]

Chez Scheme 9.5 worked on armhf. It's a bit cumbersome, because of the image bootstrapping step, but perhaps you could try to do a git bisect to find the commit that broke the arm build?

[KenDickey]

Did the "git bisect" but in ignorance ran into process problems.

Fortunately, Göran Weinholt is very good at this. He reports

3a49d0193ae57b8e31ec6a00b5b49db31a52373f

as the first bad commit.

[weinholt]

Indeed, this commit breaks arm32le:

commit 3a49d0193ae57b8e31ec6a00b5b49db31a52373f (refs/bisect/bad)
Author: dyb <dyb@scheme.com>
Date:   Fri Oct 27 23:16:47 2017 -0400

    Numerous changes to improve register/frame allocation speed for
    procedures with large numbers of variables:
    - added pass-time tracking for pre-cpnanopass passes to compile.
        compile.ss
    - added inline handler for fxdiv-and-mod
        cp0.ss, primdata.ss
    - changed order in which return-point operations are done (adjust
      sfp first, then store return values, then restore local saves) to
      avoid storing return values to homes beyond the end of the stack
      in cases where adjusting sfp might result in a call to dooverflood.
        cpnanopass.ss, np-languages.ss
    - removed unused {make-,}asm-return-registers bindings
        cpnanopass.ss
    - corrected the max-fv value field of the lambda produced by the
      hand-coded bytevector=? handler.
        cpnanopass.ss
    - reduced live-pointer and inspector free-variable mask computation
      overhead
        cpnanopass.ss
    - moved regvec cset copies to driver so they aren't copied each
      time a uvar is assigned to a register.  removed checks for
      missing register csets, since registers always have csets.
        cpnanopass.ss
    - added closure-rep else clause in record-inspector-information!.
        cpnanopass.ss
    - augmented tree representation with a constant representation
      for full trees to reduce the overhead of manipulating trees or
      subtress with all bits set.
        cpnanopass.ss
    - tree-for-each now takes start and end offsets; this cuts the
      cost of traversing and applying the action when the range of
      applicable offsets is other than 0..tree-size.
        cpnanopass.ss
    - introduced the notion of poison variables to reduce the cost of
      register/frame allocation for procedures with large sets of local
      variables.  When the number of local variables exceeds a given
      limit (currently hardwired to 1000), each variable with a large
      live range is considered poison.  A reasonable set of variables
      with large live ranges (the set of poison variables) is computed
      by successive approximation to avoid excessive overhead.  Poison
      variables directly conflict with all spillables, and all non-poison
      spillables indirectly conflict with all poison spillables through
      a shared poison-cset.  Thus poison variables cannot live in the
      same location as any other variable, i.e., they poison the location.
      Conflicts between frame locations and poison variables are handled
      normally, which allows poison variables to be assigned to
      move-related frame homes.  Poison variables are spilled prior to
      register allocation, so conflicts between registers and poison
      variables are not represented.  move relations between poison
      variables and frame variables are recorded as usual, but other
      move relations involving poison variables are not recorded.
        cpnanopass.ss, np-languages.ss
    - changed the way a uvar's degree is decremented by remove-victim!.
      instead of checking for a conflict between each pair of victim
      and keeper and decrementing when the conflict is found, remove-victim!
      now decrements the degree of each var in each victim's conflict
      set.  while this might decrement other victims' degrees unnecessarily,
      it can be much less expensive when large numbers of variables are
      involved, since the number of conflicts between two non-poison
      variables should be small due to the selection process for
      (non-)poison variables and the fact that the unspillables introduced
      by instruction selection should also have few conflicts.  That
      is, it reduces the worst-case complexity of decrementing degrees
      from O(n^2) to O(n).
        cpnanopass.ss
    - took advice in compute-degree! comment to increment the uvars in
      each registers csets rather than looping over the registers for
      each uvar asking whether the register conflicts with the uvar.
        cpnanopass.ss
    - assign-new-frame! now zeros out save-weight for local saves, since
      once they are explicitly saved and restored, they are no longer
      call-live and thus have no save cost.
        cpnanopass.ss
    - desensitized the let-values source-caching timing test slightly
        8.ms
    - updated allx, bullyx patches
        patch*

[weinholt]

Here is some more information that I hope will lead to a solution so that Chez will work on ARM again.

It appears that binary-imm-op calls + with a #f argument. I've attached some output where I inspect the error continuation: arm32bug.txt. Let me know if you need more output.

I can reproduce the compilation error Exception in +: #f is not a number consistently on commit 600edcd (current master) on arm32le with the following code. Type it into the repl or load it from a file.

(library (lib-2)
  (export x y)
  (import (rnrs))
  (define (x) '(a))
  (define (y . _) '(a)))

(library (lib-1)
  (export)
  (import (rnrs) (lib-2))
  (define a
    (y (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)))
  (define b
    (y (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x)
       (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x) (x))))

[cjfrisz]

This issue pushed me over the edge of buying a Raspberry Pi--I now have a RasPi4 4GB model running Raspbian Buster and was able to reproduce the issue. First off, thank you to @KenDickey and @weinholt for providing so much detail, as it made it relatively easy to narrow in on the source of the problem. That said, I haven't been able to find a solution, since my Chez Scheme hacking chops are a little rusty.

The error occurs because the compiler is trying to emit an "add with immediate" instruction (addi) with an invalid immediate value. On ARM, instructions using an immediate operand encode the immediate value using 12 bits: an 8-bit number with a 4-bit rotate value¹, which in the Chez Scheme codebase gets refered to as a "funky12". In arm32.ss, there is a function named funky12 (definition here) that takes an integer and returns either 1) its encoding as an ARM 12-bit immediate operand or 2) #f if the integer can't be represented as a funky12. What you're seeing when the Exception in +: #f is not a number occurs is that the code emitter for addi called funky12 with its immediate here, assumed it would get an integer back, and tried to use the result in an addition as part of emitting the output--since it got #f instead, + throws an exception.

Once I reproduced the error, I built the compiler with o=2 as @akeep recommended, plus I enabled inspector information with i=t, i.e., make o=2 i=t allx in the s directory. With the result, I dropped into the debugger and spotted that the instruction in question is essentially trying to emit (set! %lr (+ %sfp <imm>)), where <imm> = 2561 on my system, which indeed cannot be represented as a funky12.

My first thought on seeing this was "why didn't we put in a call to $oops if funky12 returned #f in binary-imm-op so it would throw a better error?" Of course, the reason that we assume the immediate will come out as a funky12 is that code generation comes after instruction selection, and one of the jobs of the instruction selector is to enforce exactly these sorts of machine constraints (though I would recommend putting in an $oops call, since clearly this can get messed up).

That indicates to me that the instruction is created or exposed after the final round of instruction selection. The breaking changeset that @weinholt identified indeed modifies behavior dealing with how things are stored on the stack (which would involve %sfp) and how function returns work (which would deal with %lr), but I haven't been able to identify anything that would get past select-instructions!. Maybe @dybvig can spot it more quickly?

¹ While I worked on tracking this down, I came across this nice blog post about 12-bit immediate operands on ARM. It both gives a good explanation and provides a handy visual tool for checking if a number can be encoded as a funky12

[KenDickey]

On 2020-02-15 13:37, Chris Frisz wrote: The error occurs because the compiler is trying to emit an "add with immediate" instruction (addi) with an invalid immediate value. ... That indicates to me that the instruction is created or exposed after the final round of instruction selection.

Great detective work! The simple solution is to pay a 1 instruction premium for 8..16 bit wide immediates. [2 instructions] E.g. [tested on Aarch64; don't have 32 bit ARM OS on hand] uint32_t word_test() { asm( "movz x0, #0xCCDD, lsl #0" ) ; /* MoveLo+Zero other bits */ asm( "movk x0, #0xAABB, lsl #16" ) ; /* MoveHi+Keep other bits */ } ==> Immediate inline 32 is 2864434397 = 0xAABBCCDD

…

-KenD [FYI: attached solution for Aarch64 inline 64 bit immediates; no PC reg available on Aarch64]

[cjfrisz]

Essentially, that's what select-instructions! should produce--it's supposed to check that the offset value can't be encoded as a funky12 and produce something like this:

(begin
  (set! <reg> #xA01)
  (set! %lr (+ %sfp <reg>))

Which should come out to something roughly like this (pardon any assembly mistakes):

mov <reg> #0xA01
add %lr %sfp <reg>

You do raise a good point: it would be an interesting experiment to disable using funky12 immediate operands for add in select-instructions!, which would force all operands into registers, and see if the error still occurs. I suspect that it will occur based on my prior hypothesis, but it will provide more evidence for that hypothesis and maybe narrow the search a little. I don't have access to my Pi at the moment, but I'll try out this test when I get a chance.

[cjfrisz]

So I've got good news and bad.

The good news is that I found the problem from the original report and the example program, and I have a fix. A bug has been lurking in this version of the ARM code since it was originally written that double-encodes the 12-bit, "funky12" immediates. It would seem that we got away with it for as long as we did because it doesn't affect numbers that fit in 8 bits since the top 4 rotate bits of the funky12 are all 0 anyway. The changeset that @weinholt identified didn't specifically change anything about instruction selection like I suspected, but it did generate code with much larger stack pointer offsets for calls to functions with a large numbers of arguments.

As I mentioned before, the immediate from the provided test program fails trying to convert the number 2561 to a funky12, which isn't possible. Examining the output from the pass before select-instructions!, finalize-frame-locations!, there is a (set! %sfp (+ %sfp 4096)), and it happens that (funky12 4096) => 2561.

The bad news is that even with my fix, there are still errors in the mats. There are at least two kinds of errors.

The first set of errors comes from calls to $read-time-stamp-counter from misc.ms. On ARM, the timestamp counter is protected and can't be read from userland without a kernel module. The code and instructions for how to load such a kernel module are available various places on the internet, but the version used for developing the ARM support for Chez Scheme is lost to the sands of time.

How to fix the $read-time-stamp-counter might require a separate discussion. On the one hand, it would be nice to have the timestamp counter available for arm32le as with the other platforms since it is possible to access it. On the other hand, I'm not sure it's a good idea for the Chez Scheme build to include checking for and/or installing dependencies for building and loading Linux kernel modules. If we don't provide the kernel module, then it seems like we should implement a compiler parameter for arm32le that specifies whether the hardware timestamp counter is available, and otherwise returns a default value or falls back to a software counter. Like I say, that probably warrants a decision from the other maintainers (\cc @akeep) and requires some additional work.

The second set of errors from the mats comes from foreign.ms, mostly from the "structs" mat. I haven't dived into it yet, but off the top of my head that indicates that there's something wrong with the alignment and/or offsets that Scheme assumes for structs. Hopefully I can resolve those pretty quickly by writing a little C code that prints out memory addresses and compare it to the offsets the compiler is using for ftypes. I also see some errors from the "varargs" mat, which may or may not get resolved with the same fix.

[weinholt]

@cjfrisz Thanks for looking into this bug; it's blocking me from uploading new Chez Scheme releases to Debian. Could you please share the fixes you have so far?

[cjfrisz]

@weinholt I apologize for the delay, and I'm sorry this is blocking you. I don't mean to let the perfect be the enemy of the good, but traditionally Chez Scheme has had a "no known issues" policy, so I'm hesitant to submit a patch that doesn't address the bugs in the mats.

I've been steadily working on the remaining issues and have a minimized test case for the issue affecting the majority of the foreign.ms mats. Plus, I have an email chain with @akeep where we've discussed how to deal with the timestamp counter. I will continue actively working on this as I have time this week, but can I hold off on making a pull request until this weekend?

[weinholt]

I will continue actively working on this as I have time this week, but can I hold off on making a pull request until this weekend?

Yes, of course, no worries whatsoever. Everything in its own time. Again, thank you much for working on this!

[cjfrisz]

I worked on this over the weekend, and the bug in the foreign.ms mats is not as straightforward as I hoped (because of course it's not). Here's a relatively minimized version of one of the test that's failing:

(begin
  (load-shared-object "./f4_sum_cb_i8.so")
  (load-shared-object "libc.so.6")
  (define-ftype callback-r (function () (& integer-8)))
  (define sum_cb (foreign-procedure "f4_sum_cb_i8"
                                    ((* callback-r)) double))
  (let ([cb (make-ftype-pointer
             callback-r
             (lambda (r)
               (ftype-set! integer-8 () r -11)))])
    (let ([v (sum_cb cb)])
      (unlock-object
       (foreign-callable-code-object
        (ftype-pointer-address cb)))
      v)))

And the corresponding C code, assumed in Scheme to be compiled to f4_sum_cb_i8.so:

double f4_sum_cb_i8 (signed char (*cb)()) {
  signed char v = cb();
  return (double)v;
}

Essentially, the test binds a foreign-procedure for f4_sum_cb_i8 to sum_cb in Scheme that expects a function pointer which takes no argument and returns a signed char/integer-8. The Scheme code creates an ftype pointer to such a procedure, cb, and hands that to f4_sum_cb_i8 via sum_cb, which invokes the function, then casts its result to a double and returns it.

I did some time in gdb, and the invalid memory reference occurs in the signed char v = cb(); line in the C code, and it looks like things go wrong while inside the Chez Scheme runtime trying to call off to the foreign-callable callback function. Specifically, something seems to go wrong in S_call_help in schlib.c when calling off to S_generic_invoke.

There are other tests for foreign-callable in foreign.ms that don't seem to fail, so it stands to reason that there is something specific to this setup with creating a foreign-callable procedure via ftype that's the problem, and not just that foreign-callables are broken. That said, I haven't tried the equivalent test without using the ftype system (i.e., using foreign-callable).

Probably the most direct way to debug this is to validate that the ftype-pointer that gets allocated has the correct shape according to the runtime. My skills for digging around in memory with gdb and figuring out the correct offsets are a little rusty, so that's the major slowdown.

I'm still reluctant to do a pull request claiming to fix the arm32le support when this is lurking in there. Are you pushing releases specifically for Debian ARM, or is there a way to leave off the ARM version until this is really fixed?

[weinholt]

I'm still reluctant to do a pull request claiming to fix the arm32le support when this is lurking in there. Are you pushing releases specifically for Debian ARM, or is there a way to leave off the ARM version until this is really fixed?

I'm pushing releases for all architectures at the same time (i386, amd64 and armhf for now; ppc was already broken, #302). I can't leave one off, but it's ok, take the time you need. A PR doesn't need to be complete from the very start though and it can help you get more eyes on the problem.

[mflatt]

The f4_sum_cb_i8 example works for my old RPi with an old Raspian, so I'm not certain what's going wrong. But the last sub-case of the (fp-ftd& ,ftd) case of do-result (which I wrote) looks suspicious:

https://github.com/cisco/ChezScheme/blob/master/s/arm32.ss#L2977

That's always reading 4 bytes from some space the stack, and maybe on an little endian machine, an 8-byte result doesn't get sign extended as it should be. It's not right at all for big-endian. Maybe the implementation there should instead be analogous to the ppc32 case:

https://github.com/cisco/ChezScheme/blob/master/s/ppc32.ss#L2984

Also, it looks like the result registers on lines 2978 and 3006 should be (list %Cretval), not (list %Cretval %r1).

The varargs failure may be because the ABI distinguishes variadic functions where FP registers should not be used for arguments. If so, Chez Scheme will need a new calling-convention option so specific varargs mode, and then the arm32 implementation can skip registers.

[cjfrisz]

Thanks for taking a look at that! I was kind of dreading the next round of debugging (and put it off for a while). This gives me some very good leads to go off of.

[mflatt]

I've spent more time on arm32le, so here are some follow-ups:

• I never saw the failure with f4_sum_cb_i8, maybe due to a forgiving gcc on my ancient Raspian installation. In an environment were it matters, though, probably signed and unsigned need to be treated differently (sign extend versus zero extend) like this: racket@158ffeb

• fix foreign-callable handling of bytevector arguments #516 addresses problems that more easily show up on arm32le, at least in my experiments, but the problem also showed up for me partly due to other changes in the Racket branch of Chez Scheme (GC changes there).

• The registers used as temporaries %flreg1 and %flreg2 are callee-save in the AArch32 ABI. Those should be saved and restored by a foreign callable. Changes to do that in the Racket branch got tangled up with changes for unboxed floating-point arithmetic, but it's the part of racket@4066af9 that creates changes in "arm.ss" starting at https://github.com/racket/ChezScheme/blob/4066af9cf3799392ef785a77da69f7cfff74d2fe/s/arm32.ss#L3307 .

• Supporting the varargs ABI on AArch32 is painful. Although I pushed that through in racket@b202943, some of the details of the commit are again related to support for unboxed floating-point, and I may have further changed something relevant in racket@4066af9. I recommend just declaring varargs unsupported and doing something about the test.

• The ftype-lock! tests failed on my machine because of a generated STREX that uses the same register for source and destination. I adjusted it this way: racket@16af343 .

[cjfrisz]

• I never saw the failure with f4_sum_cb_i8, maybe due to a forgiving gcc on my ancient Raspian installation. In an environment were it matters, though, probably signed and unsigned need to be treated differently (sign extend versus zero extend) like this: racket@158ffeb

Last I was debugging the f4_sum_cb_i8 problem, the end result was a memory reference into unallocated space. Unfortunately, it occurred after the call to Sgeneric_invoke and past the instructions for the foreign-callable prelude, so it wasn't obvious what code was executed after that without doing memory breakpoints, which I didn't make time to do. I could believe it's a sign extend vs. zero extend problem, though--I'll pull in your changes and see if that makes any difference on my more recent Raspbian.

• The ftype-lock! tests failed on my machine because of a generated STREX that uses the same register for source and destination. I adjusted it this way: racket@16af343 .

I don't have access to my notes at the moment, but if I remember correctly, I found advice for a similar error to use -fomit-frame-pointer that resolved the error for me, but I am inclined toward your solution not depending on gcc flags. It might be six of one, half dozen of the other.

[mflatt]

The -fomit-frame-pointer workaround may be related to the inline assembly that mkheader.ss generates. The inline assumbly uses r11 (the frame pointer register), but that choice seems arbitrary. I changed it to a different arbitrary register, r7, to avoid a problem compiling foreign3.c.

[cjfrisz]

Yes, that's right. At some point my changeset included an update to use -fomit-frame-pointer for building foreign.so in the mats directory. Since the register choice in mkheader.ss is arbitrary anyway, your solution is better.

[mflatt]

I'm pretty sure this is all resolved, so closing.