Line Number and Custom Backtrace lead

[mariari]

https://www.snellman.net/blog/archive/2007-12-19-pretty-sbcl-backtraces.html

This link seems to show me how to get line numbers from CL

[mariari]

I investigated some options, doing

(defparameter *x* nil)

(defun calling-from ()
  (dissect:with-truncated-stack ()
    (list (ban))))

(defun ban ()
  (again)
  3)

(defun again ()
  (setf *x*
        (make-source-test :x
                          (dissect:stack)))
  (list (dissect:stack)))

nets us with

ALU-TEST> (calling-from)
(3)
ALU-TEST> *x*
#S(SOURCE-TEST
   :X (#<DISSECT::SBCL-CALL [1] AGAIN | /home/katya/Documents/Work/repo/alu/test/global-examples.lisp:401>
       #<DISSECT::SBCL-CALL [2] BAN | /home/katya/Documents/Work/repo/alu/test/global-examples.lisp:397>
       #<DISSECT::SBCL-CALL [3] WITH-TRUNCATED-STACK-LAMBDA | /home/katya/Documents/Work/repo/alu/test/global-examples.lisp>))

which is a fairly good starting point. Note CCL gives a similar stack but non truncated.

However this has a few drawbacks:

1. this is heavy
2. Only top level definitions.

Thus it is a massive step up, as we can trace the behavior of primitives made in.

I think this can be further improved with https://github.com/s-expressionists/Eclector which provides for us a custom reader to track source information.

###Proposition

Further with dissect I propose that we compile once with it off, if the CL code throws an error of any kind, we recompile the alucard code, this time with these calls enabled. Thus our error handler now seeing this, will report the error message with a better call stack of the global functions to the instructions point.

Future Eclactor work

It seems quite nice, although I'm unsure how to change out the reader of CL, and then put this ontop

(defclass my-client (eclector.parse-result:parse-result-client)
  ())

(defmethod eclector.parse-result:make-expression-result
    ((client my-client) (result t) (children t) (source t))
  (list :result result :source source :children children))

(defmethod eclector.parse-result:make-skipped-input-result
    ((client my-client) (stream t) (reason t) (source t))
  (list :reason reason :source source))

(defparameter *y*
  (with-input-from-string (stream "(1 #|comment|# \"string\" (+ 1 2 (* 3 4)))")
    (eclector.parse-result:read (make-instance 'my-client) stream)))

(:RESULT (1 "string" (+ 1 2 (* 3 4))) :SOURCE (0 . 40) :CHILDREN
 ((:RESULT 1 :SOURCE (1 . 2) :CHILDREN NIL)
  (:REASON :BLOCK-COMMENT :SOURCE (3 . 14))
  (:RESULT "string" :SOURCE (15 . 23) :CHILDREN NIL)
  (:RESULT (+ 1 2 (* 3 4)) :SOURCE (24 . 39) :CHILDREN
   ((:RESULT + :SOURCE (25 . 26) :CHILDREN NIL)
    (:RESULT 1 :SOURCE (27 . 28) :CHILDREN NIL)
    (:RESULT 2 :SOURCE (29 . 30) :CHILDREN NIL)
    (:RESULT (* 3 4) :SOURCE (31 . 38) :CHILDREN
     ((:RESULT * :SOURCE (32 . 33) :CHILDREN NIL)
      (:RESULT 3 :SOURCE (34 . 35) :CHILDREN NIL)
      (:RESULT 4 :SOURCE (36 . 37) :CHILDREN NIL)))))))

I can probably add this source information along with the stored frozne sexp to figure out where the line numbers really are relative to the code, though the CL reader may have eaten my line numbers by then.

If I can set a top level variable that tracks the curent stack of forms that we are currently in, then I could use this as a way of debugging calls that are directly in the current form.

[mariari]

Updated plan

After much thinking about the solution, I've thought of a plan.

We utilize the fact that common lisp functions can be compiler-macros as well.

Thus we wrap every function in a

http://www.lispworks.com/documentation/HyperSpec/Body/m_define.htm

the defun of alucard will also make this.

What we will do is make a custom *stack* that holds syntax literals of the calls that have come before it.

Thus if we write.

(apply (lambda (x y) (+ x y))
       (list (* 2 3)))

then *stack* would be when we get to making the * node is.

(* 2 3)
(list (* 2 3))
(apply (lambda (x y) (+ x y))
       (list (* 2 3)))
NIL

and by the + node it looks like

(+ x y)
(lambda (x y) (+ x y))
(apply (lambda (x y) (+ x y))
       (list (* 2 3)))
NIL

for cases where the macro can't expand like

(apply #'+
       (list (* 2 3)))

then by the time we make the + node the stack will look like

(apply #'+
       (list (* 2 3)))
NIL

where the addition itself isn't recorded.

This will happen as well for any function the user makes that does not use our custom defun or they bring in from some library.

We should make a note of all functions which we've added this, so a user can flush them all if they do not want the impact on their image, or perhaps have a way for users to be able to easily tweak it via the environment, thus to not have an issue where functions defined by the user used for alucard and non alucard code does not suffer a performance hit by the macro expansion.

This technique won't give line numbers by itself, however since we know what function we are in (if we aren't in one, then it's inputed in the REPL in all likelihood!), and we've written down the literals, it's unlikely that overlap would happen N deep without that code also having the same error, as long as we have a way of also preserving the whitespace in the read syntax, then we could have precise line numbers for errors.

Future plans.

If we can figure out the line number part of the error, we should be able to figure out the call that will make sly/swank give an error that allows one's editors to jump to where the error is.

The work for this is still an unknown and will take investigation to figure out if there are calls for me to do this, and if not provide a minor editor extension for better debugging.

[mariari]

Updated Plans Again

After a prototype on #33

where I implemeneted an automatic stack pusher

(defmacro stack-compiler-macro (name)
  `(define-compiler-macro ,name (&whole form &rest args)
     "Adds pushing stack traces to the function when it can be
     expanded"
     (declare (ignorable args))
     (let ((previous-expansion *already?*))
       (setf *already?* form)
       (if (eq form previous-expansion)
           form
           `(prog2
             (push ',form)
             ,form
             (pop))))))

It works great on SBCL, but fails horribly on CCL. This is because the standard does not dictate expansion order.

However @informatimago recommend a good solution to this problem.

What if instead of trying to do all this on user expressions, instrument it in the functions I generate myself!

I thought in the past this was hard, but as he pointed out the set of special forms is fixed, as we can see in

https://www.cs.cmu.edu/Groups/AI/html/cltl/clm/node59.html

we can use (and fboundp macro-function) to detect macros, then expand them.

Then we can walk the code expanding all macros, and push to the stack as we feel.

Then we just need to interpret all the special forms and dispatch our tracing based on that.

With all that, now we can have user level tracing and it should just work.

Some notes, this means that user functions who wish to be traceable, must go through my custom defun which replaces the normal code with calls that can trace. This is an acceptable loss, as users who are generating out opcodes to Alucard likely do not care for these same functions being used in normal CL code. However we should make note of this, so that users who make general functions can ensure that I do not meddle with their expansion.

Notes and Links

Thanks @informatimago once again for the useful links.

• special forms
• stepper
  • Useful for learning how to deal with the special operators of CL
• rdp
  • Seeing how rdp handles pushing on the stack
  • Also check out line #1126

[mariari]

Work has been mostly completed, code by default is traced, all that is left is making the traces readable and noting where function boundaries are for the work I need.