Implement type system #192
Labels
Milestone
Comments
Merged
|
I have some work outstanding on this front. It will likely be Dec. 2016 before I can slice out patches to submit back upstream, however. |
|
@brpocock cool. You can if you want open a WIP pull request if you want early feedback or some help as well. |
|
It suffers from the problem of being very messily integrated with the package system fixes, “building my app, not the REPL”, etc. The branch https://github.com/romance-ii/jscl/tree/jscl-ffi-flattening has drifted quite a bit from upstream; once I have it stable enough that it doesn't hold up the game depending upon it, I can start trying to factor out reasonably-sized patches. ☹ |
brpocock
added a commit
to romance-ii/jscl
that referenced
this issue
Dec 7, 2016
> This is a very large flattening effort. This branch's goal is to > stabilize the many changes on this fork; > > 1. So that the downstream use, in Tootsville, will be able to proceed > with these advantages. > > 2. So that patches from this can be made and submitted for > upstream (JSCL-Project)'s consideration and discussion. > Hopefully many of these may ultimately be adopted upstream, so the > forks will not be so very different. > > It is not stable yet, and you should see the very end that there are > issues resolved and not. There are also many TODO / HACK /FIXME > comments, some personal (@BRFennPocock) notes tagged with ☠, and the > most noticeable, it currently doesn't even compile without crashing. > > Here are the changes, roughly grouped in a kind of sensible way, > I hope. These are abridged from many commit logs and might repeat > themselves some times. User (Compiler-running programmer) Æsthetics • Improve / add some warning messages and safety checks. Ensure that JavaScript warnings/errors will have a stack trace associated with them by throwing an |Error| object. • Some errors in JavaScript are more verbose/detailed. • Non-numeric arguments are complained-about in great detail. • out-of-range AREF on a storage vector as well. • Don't moan about unbounds in SBCL CL Yes, there are symbols defined in Common-Lisp which are neither BoundP nor FBoundP, but it's not interesting to report those in SBCL. When JSCL is the CL package, there are many more “interesting” unbounds, so we want to know about those. • Generated names in both JavaScript and Lisp code more often (almost always) use GenSym with “interesting” prefixes (and similar) to make debugging a little more intuitive. eg (let ((new-value (gensym "NEW-VALUE-"))) …) • Most messages and things use the CL tradition of leading with ~& rather than trailing with ~%. Where this was inconsistent, it was changed. • Check Situations in Eval-When for typos. • Warnings about improper behavior known to exist in DefPackage and In-Package (although these bugs may have been eliminated) • Many SBCL compile-time warnings resolved. Pulled from upstream • ANSI LOOP implementation. Some of the work done upstream to fix things for JSCL are actually “un-done” now, as the type system work, changes to macroexpansion, and other ANSI-CL-fixes made them no longer necessary. Still, I did remove some reader conditionals that might have broken the LOOP code for some pre-1990-ish compilers, so the code was not exactly returned to its pristine MIT/Symbolics implementation. • storage-vector DefStruct work — this actually was quite similar to, and merged fairly transparently with, work on this branch already. • @davazp: Add ignored size argument to make-hash-table • @davazp: Replace our mapping functions by SBCL's, so we have mapcon and mapcan for free too. Implement function NReverse. • Implement multiple-value-setq (@davazp as well I think) • @davazp had introduced his own JSIZE-SYMBOL and such, but still using the SAFE-JS-FUN-NAME in preference to new JSIZE function since they serve approximately the same purpose, but the SAFE-JS-* functions are used throughout this branch already. This will need to be harmonized at some point though. • cleanup reading invalid symbols — also @davazp or else I forget what I had meant by this. • Any number of smaller commits also Unnecessary(?) changes • DO/DO* compile from a common core • All files auto-formatted by Emacs with JSCL loaded (so macro-definitions would be known). This is rude to impose upon anyone else, and I can try to conform to individual files upstream when submitting patches, but I use the autoformatter so that I don't have to think/worry about formatting different files in different ways. It also makes my own branch merges more pleasant. • I use page breaks to navigate in Emacs and so my code is often littered with them. ANSI Common Lisp compliance • DefPackage: Handle nicknames, exports (not yet documentation, but the “manual” generator runs in SBCL) • A very poor implementation of Redefine-Package • Symbol names with too many :'s should signal a reader error. • The host's Read-Char/Peek-Char are used to enable real Gray streams on the host. JSCL's Read-Char/Peek-Char are corrected to match ANSI signatures. • LOOP (as above) — some changes here to integrate with other things as well. • Not ANSI but CLtL2: DECLARATION-INFORMATION, wanted by LOOP. Presently hard-coded to claim: (optimize '((speed 1) (debug 1) (space 1) (safety 2) (compilation-speed 1)) • FORMAT options • ~:R for ordinal numbers in English. • ~R for cardinal numbers is not really written; there exists a stub. • ~@r for Roman numerals, up to 399,999. I use the Unicode versions because they display more nicely (usually). I guess a switch to use the ASCII characters like SBCL might be useful to someone else. The highest Roman numeral defined in Unicode is ↈ (“Roman Numeral One Hundred Thousand”). Displaying a larger number is not possible. The ~:@r is interpreted the same but replaces Ⅳ with ⅠⅠⅠⅠ. It does not, however, change eg. ⅩⅬ to ⅩⅩⅩⅩ or so forth. (The only use I know of this quirk is to display time of day.) • ~( ~) should handle all four case-changing modes. • ~#\Newline should work as advertised and print nothing, not even leading spaces on the subsequent line. • ~@p and ~@:P for “-y” “-ies” should now work. • ~| → #\Page • FORMAT handles (at least simple) ~{ ~} lists. Nested ~{ ~} groups should work as well. • FORMAT ~/funcall/ • ~; and ~^ are recognized for grouping. ~{~^~} works; ~; will be used by ~[~] and ~<~> which are not yet implemented. • Define JSCL/MOP and JSCL/CLTL2 for those functions that don't belong in the Common-Lisp package. • Whitespace includes Linefeed, Return, Space, Tab, Page (Form Feed), and Vertical (Line) Tab. • Name most GenSym and GemSym-like objects so that they can be more easily traced out. • “Type System” implementation (qv, below) • Significant “standardization” of the READer to use CL functions rather than private versions (eg, PEEK-CHAR vs %PEEK-CHAR) to move toward Gray streams (or similar) • quasi-quoted (back-quoted) comma-dot should now work, but I have no practical examples to test the edge cases between comma-at and comma-dot: `(x ,. y) vs `(x ,@ y) • Cross-compilation Quasiquote handling When cross-compiling, the host Lisp (SBCL)'s reader can leave quasiquotes (backquotes) in its own internal form. This defines the backquote as though it were a macro, and binds it to an acceptable (but not public) SB-IMPL::EXPAND-QUASIQUOTE to handle rewriting the form. There's a possibility of some weirdness with this, but cross-compilation tends to be weird some days. • Treat pathnames as self-evaluating literals. Some basis for (eventually) using PATHNAME objects for URI's may have made it in. • use PARSE-BODY in macroexpander properly • FDefinition, FBoundP, FMakUnbound • Slightly safer COMMON-LISP exports list General Compiler work • Bad JavaScript detection There were cases — I think these were all fixed in the “Unicode and Radices” branch — in which invalid UTF-8 sequences or #\NULL bytes could be created by the code generator, and there are now explicit checks in compile-file.lisp to detect these cases. • optimize out some pure functions The JSCL::Pure declaration on a function is meant to mean more-or-less what a Haskell programmer or mathematician would expect: The function's outputs are precisely and always determined completely by its inputs, and there are no side-effects, including object construction. EG: arithmetic. This does primitive compile-time folding of pure functions. It should be at least smart enough to collapse (+ 2 2) → 4, and with some tagging work (floor 4 2) → (values 2 0) as well. This needs more work to be more capable, but it's a start. • Proclamation code refactored a little and PURE added for functions. • Define AND, OR, WHEN, UNLESS in terms of COND, and make COND somewhat more aggressive at removing unreachable code. COND will identify unreachable code, eliminate it, and issue a warning (during cross-compilation: a continuable error) about the unreachable cases. COND is already used for [E]CASE, so they obtain that benefit as well. These optimizations could have been done in each place individually, but it seemed to make sense to use the most general — COND — branching construct to define the others. IF is a compiler primitive, so it does not yet gain this benefit. JSCL JavaScript FFI & JS codegen • Character output to the console log in browser is buffered, and flushed by #\Return #\Newline or #\Page, or writing a subsequent string. • Define JSCL/FFI package for OGet and friends. The heart of this is to provide a useful JSCL/FFI package and support building actually usable web code from an independent build from JSCL itself. The real “test case” is in the Violet Volts mesh networking code, but there should be sufficient unit tests here to gain confidence. In the furtherance of that, some CLHS compatibility features have been added, and certain JSCL features may have been refactored into smaller pieces for either testing/debugging convenience, to enable re-use, or sometimes for æsthetics (meaning that I had trouble understanding the code until I broke it down). Some of these “fixes” are SBCL-specific. • Some fixes to prelude.js to ensure “public” JS symbols are not renamed during minification. • Magic glue for passing complex objects across the FFI boundary and remembering what Lisp type we would like them to be when they come back. Most of this code has not landed, but the idea is that when a Lisp object is converted to a JavaScript one, adding a member to that object with the extremely-unlikely-to-be-altered name of #\u+1f3db “Classical Building” (arbitrary but slightly punny choice) with the string name of the Lisp class to which we would like it converted back if it is returned to us. This does not help with objects newly constructed by JavaScript code, but it could be helpful when we allow JS to mutate a Lisp object. This code is pretty much experimental and not yet very useful. • “package” is a JS keyword (ES6?) — fix error in prelude.js that upset minifiers • use new JSCL/FFI for #j: reader • #J: reader supported in host for cross-compile — although mostly useless, it won't signal errors. • writing rationals out as ratios: so e.g. 1/3 in Lisp — ie, rational 1/3 — can be passed to JavaScript as (1/3) — ie, ( 1 ÷ 3 ) ≈ .3333333… There is some support for checking whether a rational can be expressed exactly in a float, and it doesn't actually do any fancy “denominator is a valid power of two” work, it just compares them, so there's still a possible slight rounding error. But, at least, don't complain about 1/2 being rounded to a float, because it's mostly harmless. • avoid writing Lisp floats in such a way that JavaScript can't interpret them (eg, 1.0d0) • pathname conversion to JS (poorly) The goal I have is for pathname-host to be the protocol + host + port for network streams, so #p"https://example.com:123/foo/bar.lisp" = (make-pathname :host "https://example.com:123" :directory '(:absolute "foo") :name "bar" :type "lisp" :version :newest) — this means that pathname-host matches the same-origin-policy rules. NB WebDAV has pathname-version support. Build System • Exports JSCL::Compile-Application • Some tweaks to .gitignore • Cosmetic: repl-node.js is made executable and named jscl-repl instead, to “look like” the more usual command-line program (which it is). • Some (small) amount of support for compiling with non-SBCL hosts is in place. It is mostly useless still, but it is a little effort in that direction. Stubs in the Makefile allow you to watch it fail with “LISP=$whatever make” • Merge fixes to enable building JS from within another Lisp program from SBCL (incomplete) • Consistently use in *FEATURES* :JSCL-XC for the cross-compilation of JSCL itself, and :JSCL for only when JSCL is the runtime system. IE: By examining *FEATURES*, determine whether JSCL provides Common-Lisp. • add a JSCL:Bootstrap-Core that does not build the REPLs • Replace make.sh and run-tests with a simple Makefile Upstream might not like this, but it works more nicely as a git submodule within another project • I had experimented with compile-time progress reporting and decided it was too much hassle running in different non-TTY environments (Slime, Travis CI, etc) and ripped it back out. The compile process is a bit more verbose and compiler messages now look a bit more like, eg, SBCL's — they make up a hierarchy of comments with more semicolons for higher-level messages. • Build REPLs in their own packages as well. • set *COMPILE-FILE-PATHNAME* always • Emit-Uncompilable-Form — One of the somewhat nice things about Java is that an uncompilable form can be emitted as a runtime error, in the hopes that nobody will trip over your horrible mistakes. This gives you the option of trying to proceed with compilation when things have gone wrong — it's unlikely to be a good thing in the long run, but it can sometimes be useful during rapid development. • verbose build banner Mostly just so I can keep track of where it restarted in the REPL when I'm iterating over things. • disable SBCL's debugger by default during build (for CI reasons); you can re-enable it with “LISPFLAGS= make” (blank after LISPFLAGS) • compile-file.lisp broken out from jscl.lisp and refactored into smaller pieces. May conform a little better to ANSI CL behavior also. • Tag generated files with compiler version and source files until we get Source Maps. This includes the Git revision level and there is a flag meant to be set on our branch to distinguish it from upstream. • Uniformly warn about target-only files (signal an error if they're loaded with #-JSCL-XC) This keeps me from accidentally making SBCL angry during development. It is such an instinct to recompile after editing … • When running in the host environment, if a macro-function is defined in the (host's) Common-Lisp package, and no matching macro-function is found in JSCL's, then a !-prefixed version will be used instead. This follows from jscl-project#35 but I think there is a nicer solution to replace it in future, using separate packages. This is in Check-for-Failed-MacroExpansion. Documentation • Comments in many cases were changed instead to docstrings. Some elucidations added. • Some other docstrings were added; in a couple of cases, from other compilers, particularly SBCL. • Info, PDF and HTML versions of the docstrings should now be generated by way of DECLT. Requires Quicklisp on the host compiler and Info. This is rather rudimentary still. While PDF and HTML are nice, integrating the Info with Emacs and Slime is a key goal in future. Integrating the HTML version with DESCRIBE in the Web REPL might be a fun thing also. XXX • The DECLT in Quicklisp crashed in SBCL for me, a hack to make it work is embedded in write-docs.lisp, I need to see if this should go upstream or if I've broken something locally. TODO • Very silly stub files for the manual exist for introduction, conclusion, and HTML output. • An ASDF file exists, only to show JSCL where to look for the files. It has a cloned copy of extract-version-from-package.json also. I guess we may the the first CL program to really want to write a package.json as a result of an ASDF build some day. Tests • Some tests that should work now are un-commented. • Don't run FFI tests in SBCL • Report package of unbound exports • skip (cond (1)) test This is probably a bad one to skip, but for right now may be forgiveable. FIXME • Travis CI & Circle CI YAML tweaks • The built-in “test reporting” code is significantly more verbose and has a few changes that were meant to be useful for seeking ANSI compliance. In particular, failed tests are gathered and reported at the end, and symbols that SBCL binds in the Common-Lisp package but are absent from JSCL are reported as well. These changes sacrificed, for the moment, the ability to generate HTML output correctly. This should probably be cured before offering these patches upstream. TODO • The GCL ANSI test suite is now included as a submodule, but there is not any code to actually invoke it. • The new TEST-FN from upstream was pasted into the file but not used, because it will require “rewiring” some things. TODO SetF, &Environment, and related work • Support for SetF functions in DEFUN and FDefinition and so forth. Basically, the big change that happened some time between CLtL1 and ANSI CL to allow (defun (setf foo) (new-value …) …) also nominally allows the implementation to allow (defun (X Y) (…) …). So, this batch of changes covers a few things. Firstly, FDefinition and DEFUN and Apply and FunCall (and the compiler implicit function application) should all allow a Function-Name-P form rather than only a symbol. Secondly, the process of finding whether a SetF expander function or a function bound to (SetF X) is used had to be reworked a bit. Also, I introduced some hooks that should be consistent with allowing JSCL/FFI:OGet forms in the Function-Name-P space, as a JSCL extension that I believe complies with the standard's requirements. These are not all fully implementated, but it aligns with using the #J reader macro to expand an OGet form in the CAR position, ie, (lambda (x) (#j:foo x)) → (lambda (x) ((jscl/ffi:oget *root* "foo") x) ✓ In the FSet context, this extends the compiler primitives and prelude.js to add FSet-SetF and give symbols in the package space a SetF-value; so the function (setf foo) appears as the SetF value of the symbol (intern "FOO"). I have some concerns about the integration of the FSet-type function definitions (package-interned symbols) and the Environment-type function definitions which also apply to SetFs; see the notes on those commits. This could also be extended to support ((macro-function foo) whole &optional (environment *environment*)) as SBCL more-or-less does, if that interests anyone. • (defun (setf foo) … ) defines a (block foo … ) for eg Return-From. • setf ƒ for car, cdr, nth • Support for the Nil or Global environment directly via macro-expansions. Lambda-list support for &Whole and &Environment variables. Macro-Functions should now all get the &whole and &environment parameters, where the environment defaults to jscl::*environment* when not supplied, but a new jscl::*global-environment* exists which can be accessed (as sometimes required by ANSI CL) by passing an explicit NIL environment, eg, for top-level forms. Notably, DEFUN is defined to affect the NIL environment if it is considered a top-level form, which is usually the case (even in a top-level ProgN or so forth). eg, (let ((x 42)) (defun y () (incf x))) — must bind x in the LexEnv *environment* bound by LET, but Y in the NIL (global) environment, with a reference back to X in the LexEnv. *Global-Environment* allows DEFUN to “reach up” and work this way, and a simple hack in the compiler falls back to searching the global environment when *environment* does not have a definition. This does not handle the case in which a symbol was globally redefined between the time at which *environment* was cloned from the global environment and the time at which a form in that LexEnv was compiled, which is an edge case but almost certainly a BUG. There is a conflict of representation here between the LexEnv structures used by *Environment* and the low-level FSet-type bindings in prelude.js — the resolution to this may be to remove the prelude/FSet-type code perhaps, and define packages entirely in terms of the *Global-Environment*? I haven't explored this very thoroughly, but it seems possible for the global environment to fall out-of-sync with the JavaScript definitions, and bits of the code to look in one place or the other improperly. An alternative might be to not use *global-environment* at all, but to push all global definitions up to the JavaScript symbols (for every kind of slot on the LexEnv structure). This sounds nicer, but will probably require a little more “strange” changes to the compiler-level code, and for cross-compilation, require emulation in compat.lisp to keep track of these changes. (It probably also means some interesting tables to be emitted in the “fasl” JavaScript files to populate these tables, but that is the point of the FSet declarations that now exist anyway.) • Don't allow (fdefinition '(function foo)) Pretty sure that's not a good thing. SBCL signals an error and the CLHS defines function-name as > A symbol or a list (setf symbol). We extend that for FFI with list (oget …) also, but (function symbol) isn't really a thing … It's kinda recursive, really, since you'd expect (function symbol) to be essentially a macro version of (symbol-function (quote symbol)) which in turn is (fdefinition (quote symbol)) … I've talked myself into destroying it. Type System These changes are pretty far-reaching, invasive, and more importantly 👿 👺 THIS CODE IS STILL BROKEN 🔥 👹 • Type system core: • Hierarchy of types for those types we actually support currently • Type definitions for built-in types • Compound type definitions for strings, arrays, and numeric types • DefType understanding basic type names, SATISFIES, MOD, AND, OR, MEMBER, and NOT. Does not work right with EQL or compound types like (INTEGER 0 *) yet. (VALUES) types are not handled either. • Type-Of. Returns the type name symbol, but can return a compound type form describing precisely a string, array, or integer; eg, (STRING CHARACTER 42) for a string of 42 characters, or (ARRAY T (2 3)), or (INTEGER 9 9). • Class-Of • Reworked TypeP to use the above, mostly. • Reworked TypeCase to use the new TypeP • Reworked ETypeCase slightly • Supertypes are all defined as in ANSI, but, many of the types named do not actually exist: particularly streams and conditions, but also some variant numeric types (different kinds of floats, complex, rational) and Simple-Base-String and so forth. • Some support for EQL types for DefType and (later) DefGeneric/DefMethod. • Lexical symbols can have bound to them setf-ƒ, and can name a type or a class. (They do not yet have property lists, or documentation of various types.) • “Impassible” cases in TypeCase and ETypeCase called out at compile-time. EG: (typecase 42 (number 'num) (integer 'int)) should signal a warning. Curiously, SBCL doesn't always notice the unreachable code in these cases. • Duplicated (exactly) type names issue a warning also • Numeric types that overlap in ranges (eg, (INTEGER 0 4) and (MOD 5)) are NOT yet caught. • Merge upstream branch changes to DefStruct options and structures in storage vectors from @davazp — I'm afraid the actual commits probably got merged down in the squash process along with my own. • Storage-Vectors are used for arrays, structure objects, and (eventually) other kinds of objects. The “kind” of thing is now stored alongside the underlying vector. I avoid the words “type” or “class” here to distinguish that this is a magic cookie CONS that describes what may-or-may-not be an instance of a “type” or “class” in either of Lisp or JavaScript. The definition of the “kind” is that it is a list, the FIRST of which is a general type, and the REST of which is defined in terms of that general type. For an array, the dimensions are stored here. For a structure object, the structure type is given (the layout is only known by looking up that symbol as a type). For object instances, the idea is to have a pointer to the metaobject layout (allowing layout upgrading) placed here. For now, the FIRST of KIND is of type (MEMBER ARRAY STANDARD-OBJECT STRUCTURE-OBJECT). Metaobject layouts and generic function objects might extend that list. • TypeP 'FixNumP only for numbers in the integer range of JavaScript: ±(2⁵³-1) • Handle isolation of the cases of SBCL or JSCL dealing with SBCL or JSCL-targetted code, package namespaces, and things like that. • leave early-char code in early-char, remove “incorrect” character code stuff from boot.lisp • define TypeCase in terms of TypeP • Some forward declarations between defstruct.lisp and types.lisp. This is decidedly bad “code smell” and should theoretically not be actually needed in the long run, but it does help to quash some problems for now. There is a pattern emerging that suggests that certain bits of types.lisp should be split into two modules, loaded before and after defstruct.lisp … —~— Remaining Open Issues that might not be merged into this branch. Some of these may be trivial to cure on this branch, and could “make it in.” #66 #68 #62 #53 #52 #51 #49 #47 #36 #40 #33 jscl-project#261 jscl-project#243 jscl-project#233 jscl-project#203 jscl-project#176 jscl-project#159 jscl-project#127 To be closed with this branch, but not closing Issues tickets until this branch can build and pass tests. The upstream issues of course will also need to be parcelled out into more manageable patches. #67 #64 # partial #63 #50 #48 #45 #44 #34 #30 jscl-project#259 jscl-project#258 jscl-project#228 jscl-project#192 # partial? jscl-project#164 # partial jscl-project#17 # partial —~ 𝒻𝒾𝓃𝒾𝓈 ~— fix negative hex When radix is being printed, it should appear before the sign ⇒ (let ((*print-radix* t) (*print-base* 16)) (format t "~a" -42)) #x-2A additional type check Grab the GCL ANSI tests Random puttering Still having issues with not-expanded macros and also with SBCL not liking structure-objects that were defined in the JSCL (storage-vector) way. Keep patch to ANSI-Test here. The current patch is from @JackDaniel and will probably make it into the Git version soon enough, but for the moment, I don't expect there to be “enough” things that we need to patch in the test suite itself to want to maintain a fork. More significantly, for things that are so badly broken that we have to patch the test suite (eg: perhaps a reader macro bug or something), the thing to do is to fix JSCL; in the mean time, this patch file can live as a “pebble in the sandal” so that disabled, broken tests are never forgotten completely. Ignores: renamed targets, re-organize file build docs Stabilization, debugging. Trying to get a clean build from which to close out the remaining critical issues and begin parcelling out patches to upstream. Includes some build engineering work and documentation fixes as well as tracking down build failures. two horribly stupid typos documentation mop up various compile-time errors types, macros, and format options Work on the new Type System infrastructure, and normalizing the code between hosted and self-hosting operations Some more work to get macro-expansion at compile-time working properly for cross-compilation Additional format handlers — `~r`, `~:r`, `~@r`, beginning to work on `~{ ~^ ~}` and `~[ ~]` and friends. sketch in support for other hosts ECL support more SBCL-independence fixups; quasi-quote changes did not work Ignore Emacs project-type files FORMAT ~/funcall/ operation Don't error out on forward-declared functions. This seems to be a bit more complicated than necessary, but it has to do with the awful circularity between the type system and the compiler itself. Perhaps if the low-level bindings are factored out into a quite early module, some of this can be eliminated. FSet-SetF does not neet early Eval-When Share JavaScript's MOST-{POSITIVE,NEGATIVE}-FIXNUM The compiler needs to be on the lookout for bignums, so these figures are duplicated at +MOST-{POSITIVE,NEGATIVE}-FIXNUM+ to be visible in the host compiler as well. confusion over (FUNCTION NAME) This was probably more of a problem with formatting the error message, than with the code, but to avoid the mystifying “#’NAME” messages, renamed the parameter here to FN Forward-declare functions & vars from compiler Types fixups: building hierarchy, Typecases Move all the definitions of built-in-types into a single function to establish the types and classes required by the standard (note: many/most of these, particularly conditions and CLOS things, are not actually defined to have any behavior or even be possible to make instances of, but the type names at least will be known to have the proper subtype relationships) Also trying to clean up some horrible error behaviour from [E]TypeCase, which is not yet solved at this commit Misc compiler fixes: - Fix some SetF definitions - Support typing attributes for Storage-Vectors - Move DECLARATION-INFORMATION to JSCL/CLTL2 - Be more conservative about literals that do not (yet) work in JS - Warn about bad Eval-When situations Add a no-op WITH-COMPILATION-UNIT warn unreachable subtypes in TYPECASE *** empty log message *** make declaration match the actual inferred type later Hosted macroexpansion is the bitch The semi-circular-reference hack Æsthetics of compiling DEFTYPE should EVAL BODY but this nasty hack works for a temp patch dumping function pointers … trashy hack Fix JSCL banner (violet-volts-p) out of order
brpocock
added a commit
to romance-ii/jscl
that referenced
this issue
Mar 25, 2017
warnings and such
DEFPACKAGE :EXPORT handler
define FFI package. Need to merge in from other branch.
(defpackage (:export …))
Interpret page break character as whitespace
Implement copier constructor and predicate optiosn for def!struct
Add a fake fixnum case as an integer
Add ignored size argument to make-hash-table
Replace our mapping functions by SBCL's
So we have mapcon and mapcan for free too.
Implement function NREVERSE
Define dummy subtypep function
Implement multiple-value-setq
Provide dummy CONSTANTP function
Define the 'the' special form
For now, it does nothing
Expose macroexpand-1 function
Improve stacktraces by naming lambda functions in JS
macroexpand-1 accepts optional environment argument
Provide !macroexpand function
Fix defun
The jsize-symbol function needs to generate a valid JS identifier,
otherwise the code generator will fail. So we have to be careful so we
do not print the symbols with the package prefix.
Rebase from upstream master
Also, using the SAFE-JS-FUN-NAME in preference to new JSIZE function
since they serve approximately the same purpose, but the SAFE-JS-*
functions are used throughout this branch already.
Tests for Unicode, numeric radix
Note that these will usually crash the master branch.
merge conflicts fix
Terribly stupid git merge markers made it in.
add forgotten FORMAT ~b, ~o
Fix FORMAT padding. Add docstrings.
uncomment now-working test; add a couple easy other tests
mass reformat; missing )
(a couple of close-parens got lost in the merge)
Rolling up JSCL changes — Stabilization
> This is a very large flattening effort. This branch's goal is to
> stabilize the many changes on this fork;
>
> 1. So that the downstream use, in Tootsville, will be able to proceed
> with these advantages.
>
> 2. So that patches from this can be made and submitted for
> upstream (JSCL-Project)'s consideration and discussion.
> Hopefully many of these may ultimately be adopted upstream, so the
> forks will not be so very different.
>
> It is not stable yet, and you should see the very end that there are
> issues resolved and not. There are also many TODO / HACK /FIXME
> comments, some personal (@BRFennPocock) notes tagged with ☠, and the
> most noticeable, it currently doesn't even compile without crashing.
>
> Here are the changes, roughly grouped in a kind of sensible way,
> I hope. These are abridged from many commit logs and might repeat
> themselves some times.
User (Compiler-running programmer) Æsthetics
• Improve / add some warning messages and safety checks.
Ensure that JavaScript warnings/errors will have a stack trace
associated with them by throwing an |Error| object.
• Some errors in JavaScript are more verbose/detailed.
• Non-numeric arguments are complained-about in great detail.
• out-of-range AREF on a storage vector as well.
• Don't moan about unbounds in SBCL CL
Yes, there are symbols defined in Common-Lisp which are neither
BoundP nor FBoundP, but it's not interesting to report those in SBCL.
When JSCL is the CL package, there are many more “interesting”
unbounds, so we want to know about those.
• Generated names in both JavaScript and Lisp code more often (almost
always) use GenSym with “interesting” prefixes (and similar) to make
debugging a little more intuitive. eg (let ((new-value (gensym
"NEW-VALUE-"))) …)
• Most messages and things use the CL tradition of leading with ~&
rather than trailing with ~%. Where this was inconsistent, it
was changed.
• Check Situations in Eval-When for typos.
• Warnings about improper behavior known to exist in DefPackage and
In-Package (although these bugs may have been eliminated)
• Many SBCL compile-time warnings resolved.
Pulled from upstream
• ANSI LOOP implementation. Some of the work done upstream to fix
things for JSCL are actually “un-done” now, as the type system work,
changes to macroexpansion, and other ANSI-CL-fixes made them no
longer necessary. Still, I did remove some reader conditionals that
might have broken the LOOP code for some pre-1990-ish compilers, so
the code was not exactly returned to its pristine
MIT/Symbolics implementation.
• storage-vector DefStruct work — this actually was quite similar to,
and merged fairly transparently with, work on this branch already.
• @davazp: Add ignored size argument to make-hash-table
• @davazp: Replace our mapping functions by SBCL's, so we have mapcon
and mapcan for free too. Implement function NReverse.
• Implement multiple-value-setq (@davazp as well I think)
• @davazp had introduced his own JSIZE-SYMBOL and such, but still using
the SAFE-JS-FUN-NAME in preference to new JSIZE function since they
serve approximately the same purpose, but the SAFE-JS-* functions are
used throughout this branch already. This will need to be harmonized
at some point though.
• cleanup reading invalid symbols — also @davazp or else I forget what
I had meant by this.
• Any number of smaller commits also
Unnecessary(?) changes
• DO/DO* compile from a common core
• All files auto-formatted by Emacs with JSCL loaded (so
macro-definitions would be known). This is rude to impose upon anyone
else, and I can try to conform to individual files upstream when
submitting patches, but I use the autoformatter so that I don't have
to think/worry about formatting different files in different ways.
It also makes my own branch merges more pleasant.
• I use page breaks to navigate in Emacs and so my code is often
littered with them.
ANSI Common Lisp compliance
• DefPackage: Handle nicknames, exports (not yet documentation, but the
“manual” generator runs in SBCL)
• A very poor implementation of Redefine-Package
• Symbol names with too many :'s should signal a reader error.
• The host's Read-Char/Peek-Char are used to enable real Gray streams
on the host. JSCL's Read-Char/Peek-Char are corrected to match
ANSI signatures.
• LOOP (as above) — some changes here to integrate with other things
as well.
• Not ANSI but CLtL2: DECLARATION-INFORMATION, wanted by LOOP.
Presently hard-coded to claim:
(optimize '((speed 1) (debug 1) (space 1)
(safety 2) (compilation-speed 1))
• FORMAT options
• ~:R for ordinal numbers in English.
• ~R for cardinal numbers is not really written; there exists a stub.
• ~@R for Roman numerals, up to 399,999.
I use the Unicode versions because they display more
nicely (usually). I guess a switch to use the ASCII characters
like SBCL might be useful to someone else.
The highest Roman numeral defined in Unicode is ↈ (“Roman Numeral
One Hundred Thousand”). Displaying a larger number is
not possible.
The ~:@R is interpreted the same but replaces Ⅳ with ⅠⅠⅠⅠ. It does
not, however, change eg. ⅩⅬ to ⅩⅩⅩⅩ or so forth. (The only use
I know of this quirk is to display time of day.)
• ~( ~) should handle all four case-changing modes.
• ~#\Newline should work as advertised and print nothing, not even
leading spaces on the subsequent line.
• ~@P and ~@:P for “-y” “-ies” should now work.
• ~| → #\Page
• FORMAT handles (at least simple) ~{ ~} lists. Nested ~{ ~} groups
should work as well.
• FORMAT ~/funcall/
• ~; and ~^ are recognized for grouping. ~{~^~} works; ~; will be
used by ~[~] and ~<~> which are not yet implemented.
• Define JSCL/MOP and JSCL/CLTL2 for those functions that don't belong
in the Common-Lisp package.
• Whitespace includes Linefeed, Return, Space, Tab, Page (Form Feed),
and Vertical (Line) Tab.
• Name most GenSym and GemSym-like objects so that they can be more
easily traced out.
• “Type System” implementation (qv, below)
• Significant “standardization” of the READer to use CL functions
rather than private versions (eg, PEEK-CHAR vs %PEEK-CHAR) to move
toward Gray streams (or similar)
• quasi-quoted (back-quoted) comma-dot should now work, but I have no
practical examples to test the edge cases between comma-at and
comma-dot: `(x ,. y) vs `(x ,@ y)
• Cross-compilation Quasiquote handling
When cross-compiling, the host Lisp (SBCL)'s reader can leave
quasiquotes (backquotes) in its own internal form. This defines the
backquote as though it were a macro, and binds it to an
acceptable (but not public) SB-IMPL::EXPAND-QUASIQUOTE to handle
rewriting the form. There's a possibility of some weirdness with
this, but cross-compilation tends to be weird some days.
• Treat pathnames as self-evaluating literals. Some basis
for (eventually) using PATHNAME objects for URI's may have made
it in.
• use PARSE-BODY in macroexpander properly
• FDefinition, FBoundP, FMakUnbound
• Slightly safer COMMON-LISP exports list
General Compiler work
• Bad JavaScript detection
There were cases — I think these were all fixed in the “Unicode and
Radices” branch — in which invalid UTF-8 sequences or #\NULL bytes
could be created by the code generator, and there are now explicit
checks in compile-file.lisp to detect these cases.
• optimize out some pure functions
The JSCL::Pure declaration on a function is meant to mean
more-or-less what a Haskell programmer or mathematician would expect:
The function's outputs are precisely and always determined completely
by its inputs, and there are no side-effects, including object
construction. EG: arithmetic. This does primitive compile-time
folding of pure functions. It should be at least smart enough to
collapse (+ 2 2) → 4, and with some tagging work (floor 4 2)
→ (values 2 0) as well.
This needs more work to be more capable, but it's a start.
• Proclamation code refactored a little and PURE added for functions.
• Define AND, OR, WHEN, UNLESS in terms of COND, and make COND somewhat
more aggressive at removing unreachable code.
COND will identify unreachable code, eliminate it, and issue
a warning (during cross-compilation: a continuable error) about the
unreachable cases. COND is already used for [E]CASE, so they obtain
that benefit as well.
These optimizations could have been done in each place individually,
but it seemed to make sense to use the most general — COND —
branching construct to define the others.
IF is a compiler primitive, so it does not yet gain this benefit.
JSCL JavaScript FFI & JS codegen
• Character output to the console log in browser is buffered, and
flushed by #\Return #\Newline or #\Page, or writing
a subsequent string.
• Define JSCL/FFI package for OGet and friends.
The heart of this is to provide a useful JSCL/FFI package and support
building actually usable web code from an independent build from JSCL
itself. The real “test case” is in the Violet Volts mesh networking
code, but there should be sufficient unit tests here to
gain confidence.
In the furtherance of that, some CLHS compatibility features have
been added, and certain JSCL features may have been refactored into
smaller pieces for either testing/debugging convenience, to enable
re-use, or sometimes for æsthetics (meaning that I had trouble
understanding the code until I broke it down).
Some of these “fixes” are SBCL-specific.
• Some fixes to prelude.js to ensure “public” JS symbols are not
renamed during minification.
• Magic glue for passing complex objects across the FFI boundary and
remembering what Lisp type we would like them to be when they
come back.
Most of this code has not landed, but the idea is that when a Lisp
object is converted to a JavaScript one, adding a member to that
object with the extremely-unlikely-to-be-altered name of #\u+1f3db
“Classical Building” (arbitrary but slightly punny choice) with the
string name of the Lisp class to which we would like it converted
back if it is returned to us.
This does not help with objects newly constructed by JavaScript code,
but it could be helpful when we allow JS to mutate a Lisp object.
This code is pretty much experimental and not yet very useful.
• “package” is a JS keyword (ES6?) — fix error in prelude.js that upset
minifiers
• use new JSCL/FFI for #j: reader
• #J: reader supported in host for cross-compile — although mostly
useless, it won't signal errors.
• writing rationals out as ratios: so e.g. 1/3 in Lisp — ie, rational
1/3 — can be passed to JavaScript as (1/3) — ie, ( 1 ÷ 3 ) ≈
.3333333…
There is some support for checking whether a rational can be
expressed exactly in a float, and it doesn't actually do any fancy
“denominator is a valid power of two” work, it just compares them, so
there's still a possible slight rounding error. But, at least, don't
complain about 1/2 being rounded to a float, because it's
mostly harmless.
• avoid writing Lisp floats in such a way that JavaScript can't
interpret them (eg, 1.0d0)
• pathname conversion to JS (poorly)
The goal I have is for pathname-host to be the protocol + host + port
for network streams, so #p"https://example.com:123/foo/bar.lisp"
= (make-pathname :host "https://example.com:123" :directory
'(:absolute "foo") :name "bar" :type "lisp" :version :newest) — this
means that pathname-host matches the same-origin-policy rules.
NB WebDAV has pathname-version support.
Build System
• Exports JSCL::Compile-Application
• Some tweaks to .gitignore
• Cosmetic: repl-node.js is made executable and named jscl-repl
instead, to “look like” the more usual command-line program (which it
is).
• Some (small) amount of support for compiling with non-SBCL hosts is
in place. It is mostly useless still, but it is a little effort in
that direction. Stubs in the Makefile allow you to watch it fail with
“LISP=$whatever make”
• Merge fixes to enable building JS from within another Lisp program
from SBCL (incomplete)
• Consistently use in *FEATURES* :JSCL-XC for the cross-compilation of
JSCL itself, and :JSCL for only when JSCL is the runtime system. IE:
By examining *FEATURES*, determine whether JSCL provides Common-Lisp.
• add a JSCL:Bootstrap-Core that does not build the REPLs
• Replace make.sh and run-tests with a simple Makefile
Upstream might not like this, but it works more nicely as a git
submodule within another project
• I had experimented with compile-time progress reporting and decided
it was too much hassle running in different non-TTY
environments (Slime, Travis CI, etc) and ripped it back out.
The compile process is a bit more verbose and compiler messages now
look a bit more like, eg, SBCL's — they make up a hierarchy of
comments with more semicolons for higher-level messages.
• Build REPLs in their own packages as well.
• set *COMPILE-FILE-PATHNAME* always
• Emit-Uncompilable-Form — One of the somewhat nice things about Java
is that an uncompilable form can be emitted as a runtime error, in
the hopes that nobody will trip over your horrible mistakes.
This gives you the option of trying to proceed with compilation when
things have gone wrong — it's unlikely to be a good thing in the long
run, but it can sometimes be useful during rapid development.
• verbose build banner
Mostly just so I can keep track of where it restarted in the REPL
when I'm iterating over things.
• disable SBCL's debugger by default during build (for CI reasons); you
can re-enable it with “LISPFLAGS= make” (blank after LISPFLAGS)
• compile-file.lisp broken out from jscl.lisp and refactored into
smaller pieces. May conform a little better to ANSI CL behavior also.
• Tag generated files with compiler version and source files until we
get Source Maps. This includes the Git revision level and there is
a flag meant to be set on our branch to distinguish it from upstream.
• Uniformly warn about target-only files (signal an error if they're
loaded with #-JSCL-XC)
This keeps me from accidentally making SBCL angry during development.
It is such an instinct to recompile after editing …
• When running in the host environment, if a macro-function is defined
in the (host's) Common-Lisp package, and no matching macro-function
is found in JSCL's, then a !-prefixed version will be used instead.
This follows from https://github.com/jscl-project/jscl/issues/35 but
I think there is a nicer solution to replace it in future, using
separate packages.
This is in Check-for-Failed-MacroExpansion.
Documentation
• Comments in many cases were changed instead to docstrings.
Some elucidations added.
• Some other docstrings were added; in a couple of cases, from other
compilers, particularly SBCL.
• Info, PDF and HTML versions of the docstrings should now be generated
by way of DECLT. Requires Quicklisp on the host compiler and Info.
This is rather rudimentary still. While PDF and HTML are nice,
integrating the Info with Emacs and Slime is a key goal in future.
Integrating the HTML version with DESCRIBE in the Web REPL might be
a fun thing also. XXX
• The DECLT in Quicklisp crashed in SBCL for me, a hack to make it work
is embedded in write-docs.lisp, I need to see if this should go
upstream or if I've broken something locally. TODO
• Very silly stub files for the manual exist for introduction,
conclusion, and HTML output.
• An ASDF file exists, only to show JSCL where to look for the files.
It has a cloned copy of extract-version-from-package.json also.
I guess we may the the first CL program to really want to write
a package.json as a result of an ASDF build some day.
Tests
• Some tests that should work now are un-commented.
• Don't run FFI tests in SBCL
• Report package of unbound exports
• skip (cond (1)) test
This is probably a bad one to skip, but for right now may
be forgiveable. FIXME
• Travis CI & Circle CI YAML tweaks
• The built-in “test reporting” code is significantly more verbose and
has a few changes that were meant to be useful for seeking
ANSI compliance.
In particular, failed tests are gathered and reported at the end, and
symbols that SBCL binds in the Common-Lisp package but are absent
from JSCL are reported as well.
These changes sacrificed, for the moment, the ability to generate
HTML output correctly. This should probably be cured before offering
these patches upstream. TODO
• The GCL ANSI test suite is now included as a submodule, but there is
not any code to actually invoke it.
• The new TEST-FN from upstream was pasted into the file but not used,
because it will require “rewiring” some things. TODO
SetF, &Environment, and related work
• Support for SetF functions in DEFUN and FDefinition and so forth.
Basically, the big change that happened some time between CLtL1 and
ANSI CL to allow (defun (setf foo) (new-value …) …) also nominally
allows the implementation to allow (defun (X Y) (…) …). So, this
batch of changes covers a few things.
Firstly, FDefinition and DEFUN and Apply and FunCall (and the
compiler implicit function application) should all allow
a Function-Name-P form rather than only a symbol.
Secondly, the process of finding whether a SetF expander function or
a function bound to (SetF X) is used had to be reworked a bit.
Also, I introduced some hooks that should be consistent with allowing
JSCL/FFI:OGet forms in the Function-Name-P space, as a JSCL extension
that I believe complies with the standard's requirements. These are
not all fully implementated, but it aligns with using the #J reader
macro to expand an OGet form in the CAR position,
ie, (lambda (x) (#j:foo x)) → (lambda (x) ((jscl/ffi:oget *root*
"foo") x) ✓
In the FSet context, this extends the compiler primitives and
prelude.js to add FSet-SetF and give symbols in the package space
a SetF-value; so the function (setf foo) appears as the SetF value of
the symbol (intern "FOO").
I have some concerns about the integration of the FSet-type function
definitions (package-interned symbols) and the Environment-type
function definitions which also apply to SetFs; see the notes on
those commits.
This could also be extended to support ((macro-function foo) whole
&optional (environment *environment*)) as SBCL more-or-less does, if
that interests anyone.
• (defun (setf foo) … ) defines a (block foo … ) for eg Return-From.
• setf ƒ for car, cdr, nth
• Support for the Nil or Global environment directly via
macro-expansions. Lambda-list support for &Whole and
&Environment variables.
Macro-Functions should now all get the &whole and &environment
parameters, where the environment defaults to jscl::*environment*
when not supplied, but a new jscl::*global-environment* exists which
can be accessed (as sometimes required by ANSI CL) by passing an
explicit NIL environment, eg, for top-level forms. Notably, DEFUN is
defined to affect the NIL environment if it is considered a top-level
form, which is usually the case (even in a top-level ProgN or so
forth).
eg, (let ((x 42)) (defun y () (incf x))) — must bind x in
the LexEnv *environment* bound by LET, but Y in the NIL (global)
environment, with a reference back to X in the
LexEnv. *Global-Environment* allows DEFUN to “reach up” and work this
way, and a simple hack in the compiler falls back to searching the
global environment when *environment* does not have a definition.
This does not handle the case in which a symbol was globally
redefined between the time at which *environment* was cloned from the
global environment and the time at which a form in that LexEnv was
compiled, which is an edge case but almost certainly a BUG.
There is a conflict of representation here between the LexEnv
structures used by *Environment* and the low-level FSet-type bindings
in prelude.js — the resolution to this may be to remove the
prelude/FSet-type code perhaps, and define packages entirely in terms
of the *Global-Environment*? I haven't explored this very thoroughly,
but it seems possible for the global environment to fall out-of-sync
with the JavaScript definitions, and bits of the code to look in one
place or the other improperly.
An alternative might be to not use *global-environment* at all, but
to push all global definitions up to the JavaScript symbols (for
every kind of slot on the LexEnv structure). This sounds nicer, but
will probably require a little more “strange” changes to the
compiler-level code, and for cross-compilation, require emulation in
compat.lisp to keep track of these changes. (It probably also means
some interesting tables to be emitted in the “fasl” JavaScript files
to populate these tables, but that is the point of the FSet
declarations that now exist anyway.)
• Don't allow (fdefinition '(function foo))
Pretty sure that's not a good thing. SBCL signals an error and the
CLHS defines function-name as
> A symbol or a list (setf symbol).
We extend that for FFI with list (oget …) also, but (function symbol)
isn't really a thing …
It's kinda recursive, really, since you'd expect (function symbol) to
be essentially a macro version of (symbol-function (quote symbol))
which in turn is (fdefinition (quote symbol)) …
I've talked myself into destroying it.
Type System
These changes are pretty far-reaching, invasive, and more
importantly
👿 👺 THIS CODE IS STILL BROKEN 🔥 👹
• Type system core:
• Hierarchy of types for those types we actually support currently
• Type definitions for built-in types
• Compound type definitions for strings, arrays, and numeric types
• DefType understanding basic type names, SATISFIES, MOD, AND, OR,
MEMBER, and NOT. Does not work right with EQL or compound types
like (INTEGER 0 *) yet. (VALUES) types are not handled either.
• Type-Of. Returns the type name symbol, but can return a compound
type form describing precisely a string, array, or integer;
eg, (STRING CHARACTER 42) for a string of 42 characters, or (ARRAY
T (2 3)), or (INTEGER 9 9).
• Class-Of
• Reworked TypeP to use the above, mostly.
• Reworked TypeCase to use the new TypeP
• Reworked ETypeCase slightly
• Supertypes are all defined as in ANSI, but, many of the types named
do not actually exist: particularly streams and conditions, but also
some variant numeric types (different kinds of floats, complex,
rational) and Simple-Base-String and so forth.
• Some support for EQL types for DefType and (later) DefGeneric/DefMethod.
• Lexical symbols can have bound to them setf-ƒ, and can name a type or
a class. (They do not yet have property lists, or documentation of
various types.)
• “Impassible” cases in TypeCase and ETypeCase called out at
compile-time. EG: (typecase 42 (number 'num) (integer 'int)) should
signal a warning.
Curiously, SBCL doesn't always notice the unreachable code in
these cases.
• Duplicated (exactly) type names issue a warning also
• Numeric types that overlap in ranges (eg, (INTEGER 0 4) and (MOD 5))
are NOT yet caught.
• Merge upstream branch changes to DefStruct options and structures in
storage vectors from @davazp — I'm afraid the actual commits probably
got merged down in the squash process along with my own.
• Storage-Vectors are used for arrays, structure objects,
and (eventually) other kinds of objects. The “kind” of thing is now
stored alongside the underlying vector. I avoid the words “type” or
“class” here to distinguish that this is a magic cookie CONS that
describes what may-or-may-not be an instance of a “type” or “class”
in either of Lisp or JavaScript.
The definition of the “kind” is that it is a list, the FIRST of which
is a general type, and the REST of which is defined in terms of that
general type.
For an array, the dimensions are stored here. For a structure object,
the structure type is given (the layout is only known by looking up
that symbol as a type). For object instances, the idea is to have
a pointer to the metaobject layout (allowing layout upgrading) placed
here.
For now, the FIRST of KIND is of type (MEMBER ARRAY STANDARD-OBJECT
STRUCTURE-OBJECT). Metaobject layouts and generic function objects
might extend that list.
• TypeP 'FixNumP only for numbers in the integer range of JavaScript:
±(2⁵³-1)
• Handle isolation of the cases of SBCL or JSCL dealing with SBCL or
JSCL-targetted code, package namespaces, and things like that.
• leave early-char code in early-char, remove “incorrect” character
code stuff from boot.lisp
• define TypeCase in terms of TypeP
• Some forward declarations between defstruct.lisp and types.lisp.
This is decidedly bad “code smell” and should theoretically not be
actually needed in the long run, but it does help to quash some
problems for now. There is a pattern emerging that suggests that
certain bits of types.lisp should be split into two modules, loaded
before and after defstruct.lisp …
—~—
Remaining Open Issues that might not be merged into this branch. Some of
these may be trivial to cure on this branch, and could “make it in.”
https://github.com/romance-ii/jscl/issues/66
https://github.com/romance-ii/jscl/issues/68
https://github.com/romance-ii/jscl/issues/62
https://github.com/romance-ii/jscl/issues/53
https://github.com/romance-ii/jscl/issues/52
https://github.com/romance-ii/jscl/issues/51
https://github.com/romance-ii/jscl/issues/49
https://github.com/romance-ii/jscl/issues/47
https://github.com/romance-ii/jscl/issues/36
https://github.com/romance-ii/jscl/issues/40
https://github.com/romance-ii/jscl/issues/33
https://github.com/jscl-project/jscl/issues/261
https://github.com/jscl-project/jscl/issues/243
https://github.com/jscl-project/jscl/issues/233
https://github.com/jscl-project/jscl/issues/203
https://github.com/jscl-project/jscl/issues/176
https://github.com/jscl-project/jscl/issues/159
https://github.com/jscl-project/jscl/issues/127
To be closed with this branch, but not closing Issues tickets until this
branch can build and pass tests. The upstream issues of course will also
need to be parcelled out into more manageable patches.
https://github.com/romance-ii/jscl/issues/67
https://github.com/romance-ii/jscl/issues/64 # partial
https://github.com/romance-ii/jscl/issues/63
https://github.com/romance-ii/jscl/issues/50
https://github.com/romance-ii/jscl/issues/48
https://github.com/romance-ii/jscl/issues/45
https://github.com/romance-ii/jscl/issues/44
https://github.com/romance-ii/jscl/issues/34
https://github.com/romance-ii/jscl/issues/30
https://github.com/jscl-project/jscl/issues/259
https://github.com/jscl-project/jscl/issues/258
https://github.com/jscl-project/jscl/issues/228
https://github.com/jscl-project/jscl/issues/192 # partial?
https://github.com/jscl-project/jscl/issues/164 # partial
https://github.com/jscl-project/jscl/issues/17 # partial
—~ 𝒻𝒾𝓃𝒾𝓈 ~—
fix negative hex
When radix is being printed, it should appear before the sign
⇒ (let ((*print-radix* t) (*print-base* 16)) (format t "~a" -42))
#x-2A
additional type check
Grab the GCL ANSI tests
Random puttering
Still having issues with not-expanded macros and
also with SBCL not liking structure-objects that
were defined in the JSCL (storage-vector) way.
Keep patch to ANSI-Test here.
The current patch is from @jackdaniel and will probably make it into the
Git version soon enough, but for the moment, I don't expect there to be
“enough” things that we need to patch in the test suite itself to want
to maintain a fork.
More significantly, for things that are so badly broken that we have to
patch the test suite (eg: perhaps a reader macro bug or something), the
thing to do is to fix JSCL; in the mean time, this patch file can live
as a “pebble in the sandal” so that disabled, broken tests are never
forgotten completely.
Ignores: renamed targets, re-organize file
build docs
Stabilization, debugging.
Trying to get a clean build from which to close out
the remaining critical issues and begin parcelling out
patches to upstream.
Includes some build engineering work and documentation
fixes as well as tracking down build failures.
two horribly stupid typos
documentation
mop up various compile-time errors
types, macros, and format options
Work on the new Type System infrastructure, and normalizing the code
between hosted and self-hosting operations
Some more work to get macro-expansion at compile-time working properly
for cross-compilation
Additional format handlers — `~r`, `~:r`, `~@r`, beginning to work on
`~{ ~^ ~}` and `~[ ~]` and friends.
sketch in support for other hosts
ECL support
more SBCL-independence
fixups; quasi-quote changes did not work
Ignore Emacs project-type files
FORMAT ~/funcall/ operation
Don't error out on forward-declared functions.
This seems to be a bit more complicated than necessary, but it
has to do with the awful circularity between the type system
and the compiler itself. Perhaps if the low-level bindings are
factored out into a quite early module, some of this can be
eliminated.
FSet-SetF does not neet early Eval-When
Share JavaScript's MOST-{POSITIVE,NEGATIVE}-FIXNUM
The compiler needs to be on the lookout for bignums, so these
figures are duplicated at +MOST-{POSITIVE,NEGATIVE}-FIXNUM+
to be visible in the host compiler as well.
confusion over (FUNCTION NAME)
This was probably more of a problem with formatting
the error message, than with the code, but to avoid
the mystifying “#’NAME” messages, renamed the parameter
here to FN
Forward-declare functions & vars from compiler
Types fixups: building hierarchy, Typecases
Move all the definitions of built-in-types into a single function
to establish the types and classes required by the standard
(note: many/most of these, particularly conditions and CLOS things,
are not actually defined to have any behavior or even be possible
to make instances of, but the type names at least will be known
to have the proper subtype relationships)
Also trying to clean up some horrible error behaviour from
[E]TypeCase, which is not yet solved at this commit
Misc compiler fixes:
- Fix some SetF definitions
- Support typing attributes for Storage-Vectors
- Move DECLARATION-INFORMATION to JSCL/CLTL2
- Be more conservative about literals that do not (yet) work in JS
- Warn about bad Eval-When situations
Add a no-op WITH-COMPILATION-UNIT
warn unreachable subtypes in TYPECASE
*** empty log message ***
make declaration match the actual inferred type later
Hosted macroexpansion is the bitch
The semi-circular-reference hack
Æsthetics of compiling
DEFTYPE should EVAL BODY
but this nasty hack works for a temp patch
dumping function pointers … trashy hack
Fix JSCL banner (violet-volts-p) out of order
merge failure
remove git garbage
&environment fall-out
make sure environment is set properly
fix quasi-quote-removal for SBCL to take (ignored) environment arg
defstruct: regression: strings
Need symbol, not string, in checking struct
accessor objects and such
reformat
JSCL/COMMON-LISP package (step 1)
JSCL/COMMON-LISP package (step 2)
JavaScript emulation insanity.
The JSCL/CL madness now survives the first compiler pass.
need KEYWORD before even COMMON-LISP
JSCL/CL package; CONCATENATE
Move to JSCL/CL package
Also fix CONCATENATE to be a little bit closer to ANSI (handle a few
cases that it could not before) — still much to be done.
JSCL/CL; VECTOR-PUSH-EXTEND fixes
Move to JSCL/CL package
A few fixes (untested) for VECTOR-PUSH-EXTEND to get a bit closer to
ANSI requirements
JSCL/CL and fake generic functions
Paving the way for Gray streams implementation using some very shoddy
fake generic functions.
Actually, this is more like a C++ “virtual dispatch table” (vtbl) here,
but it should™ pave the way for proper Gray streams once DefGeneric is
in place.
Console output class
Partial implementation of the web console output as a “class” of stream
using the cheezy fake generic functions.
Needs to be properly “subclassed” by hacking it into the type system,
and the REPL's will also need to be patched to work like this.
Work-in-progress.
fixups for web-console-output-stream
missing arg
JSCL/CL and title-case char names
Breaks with SBCL's method of printing char names in UPPER-CASE but
matches with the preference Mr Steele offers (and I prefer) in CLtL2 —
char names like "Latin_Capital_Letter_L" rather than
"LATIN_CAPITAL_LETTER_L". They're case-insensitive, regardless.
Also switched to the easier-to-follow hex values for Unicode ranges now
that we have (for some time now had) #x reader “macro” (actually still
a hard-coded hack) support.
build more modules in :both
define vars and cons functions
Just maps car, cdr, rplaca, etc to the host's versions. Not re-inventing
that particular wheel if we don't have to.
Also trying to recycle a Ruby compiler to perform “var lifting” such
that something like:
function () {
blah();
var x = foo();
blah(x);
}
… will be transformed into …
(lambda ()
(let (x)
(blah)
(setq x (foo))
(blah x)))
Probably needs some work on real JS scoping rules, but might work well
enough for our generated code.
fix up JSCL/CL revisions
CL Compile-File
JSCL/CL renaming and JSCL/JS emulation
This branch's problems with weird naming conflicts led to my moving
forward with this bit of bootstrapping madness.
The changes now in progress here are to create a couple of new packages,
which have lives of their own.
First Pass:
Most of JSCL will first be compiled into JSCL/HOSTED, which uses the
code in compat.lisp (JSCL/JS functions) to emulate some of the
behaviors of the code generator. In other words, we pass through the
compiler to the same JSCL/JS low-level primitives that we would use to
generate JavaScript code — but, then, we stop short, and instead
populate JSCL/HOSTED.
The source code refers only to the JSCL/CL package. At the time this
first pass runs, JSCL/CL is JSCL/HOSTED.
Second Pass:
Next, we invoke the hosted code, and use it to read back in its own
source code — only, we don't load the compatibility layer, and we *do*
load the code generator.
At this point, we are reading in and generating code, still aimed at
a JSCL/CL package.
Pivot:
Once that JavaScript code has been written, we'll “pivot” the package
name, making the JSCL/CL package now the COMMON-LISP package.
(Possible): Refined self-hosting pass?:
In order to clear out any debris from the hosted pass, it might also
be wise to run (via Node.js, presumably) a “self-hosted” pure (third)
pass, in which the source code is (once again) recompiled by JSCL, but
now running natively.
If nothing else, the quine-ness of it should be a nice
proof-of-concept (and who doesn't love a self-hosting compiler?)
Realistically … there may be more Node.js installations in the world
than SBCL right now … so the initial bootstrap phase could end up
being of less use than the self-hosted version …
At this point, much of the work to survive the first pass has been done.
The rest remains …
misc fiddling
doc.css
Fix loading FASLs during first-stage bootstrap
Load “misc” nearer the end; nothing (much?) depends upon it
Fix jscl-repl name to match
JSCL/JS emulate MAP-FOR-IN with MAPHASH
JSCL/JS fix IN, add NEW
Order of operands of JSCL/JS::IN were reversed
Clean up symbol-name-printing function slightly
Also move some comments into docstrings
Clean up compiler errors
Moved CL symbol exports table to boot.lisp
“You should have received a copy…” Oops! here it is
Fix up names for auto-credits
UI/cosmetic changes for credits, &c
Gather git commit; present credits; comply with the GPL
“advertising” requirements for “interactive mode” in the REPL
toplevel.
Also trap and print errors in EVAL-INTERACTIVE (pending any
debugger)
Unicode typography
Don't be stupid finding src-dir
Until we get to ASDF, it's a pain to find the source-dir, but these
functions were getting totally confused when Slime ran things in /tmp
from hitting <Execute> on individual functions, because *load-pathname*
would be set "/tmp/" and we'd do things like trying to get git revisions
of your temp-dir (fun…) or read the GPL text from there (who knows,
maybe someone left a copy there…)
♯J improvements; use reader macro function in JSCL reader
In this patch, moved the CL (compat.lisp) ♯J reader into read.lisp,
where it can now be used in both models (SBCL/CL:READ or JSCL/CL:READ).
(NOTE: Using actual sharp-sign ♯ here so Git will not treat the
octothorpe number-sign as a comment marker.)
The following is the revised documentation on the WITH-SHARP-J macro,
which is the best description right now. Note the new comma/at-sign
forms described below.
————————————————————
The ♯J macro for reading JavaScript forms:
♯J:a:b:c is equivalent to (JSCL/FFI:OGET* *ROOT* \"a\" \"b\" \"c\"),
which in turn is equivalent to the JavaScript
window[\"a\"][\"b\"][\"c\"] or (when the keys are valid identifiers)
also the same as window.a.b.c and a.b.c.
Note that the ♯J: sequence is \emph{case-sensitive} and begins with
a colon (:) after the ♯J. Each :-delimited substring is the name of
a key on the preceding object in the sequence.
TODO: An equivalent form where the reference object does not have to be
*ROOT* or where identifiers may not have to be strings. There is
a discussion in http://github.com/jscl-project/jscl/Issues/
An experimental version is incorporated here. In this notation:
Begin with “♯J”. Follow this with : to refer to the root object (normally
“window”). Follow it instead with @ to NOT refer to the root object, but
instead, name an object (locally in scope) to be referenced.
For example, ♯J@foo is identical to FOO.
Each subsequent section is introduced by either a colon or comma.
A section introduced by : is treated as a string name. A section
introduced by a , is instead the name of a locally available (current
package) symbol, which is bound to the string value to be used as a key.
Examples:
♯J:foo:bar → window[\"foo\"][\"bar\"]
♯J@foo:bar → foo[\"bar\"]
♯J:foo,bar → window[\"foo\"][ bar ]
♯J@foo,bar → foo[ bar ]
Note that @ in the first position BOTH suppresses the automatic use of
the root object (window) AND ALSO acts like a comma, “unquoting” the
following argument.
Note that the @ and , options are EXPERIMENTAL and local to the Romance
Ⅱbranch. The discussion upstream may substantially change the meaning of
these options.
Rationale: @ and , are used in Lisp as quasi-quote operators, and are
not valid JavaScript identifier components, so they are unlikely to
occur in either language in normal use.
NOT-TMP: Don't choke on NIL pathnames
SRC-DIR: fix sloppy code
Handle emulating JS “new” and “this” better
Improve MAKE-NEW to invoke constructors
Bind JSCL/JS::THIS as a dynamic variable for JS “this” (or should it be
literally JSCL/JS::|this|?)
Define a p-list-based constructor for |Object|, which is probably the
wrong thing to do, but time will tell soon enough.
Fixes to ♯J reader code
Split up compatibility code for bootstrap
The compat.lisp code was getting kinda big. Splitting it up into four
separate files just for maintenance purposes.
Move some things back from compile-file
Collecting some utility functions back in jscl.lisp; perhaps these all
belong in some kind of utility file.
Some of them can later be factored out in favour of Quicklisp libraries or
even just UIOP functions.
ignore VAR (we know it's JSCL/JS:VAR)
add OSET* to compat-ffi
Clean up LOAD-JSCL merge
must load compat-sv also
unused var removed
ignore unused param
Fix JSCL/CL package for types.lisp
intern type symbols in JSCL/CL
JSCL/CL in return-from
also one random bug in STRING/= (s2 for n2)
CHECK-TYPE: string name for compound types
Until we can pretty-print it better, compound types can
at least display as their form (eg: (AND INTEGER FIXNUM))
Previously signaled an error trying to coerce the list
to a string directly
Wishlist XXX — actually render a nice description
remove stray symbol
Fix bug in ♯J reader
String & Object ctors in *ROOT*
package names on Return-From's
Same code, emulation or not
formatting only
Re-order compat and try to load each FASL
remove Define-CL-Fun finally
convert J-Reader to Loop syntax
Ran into some quirkiness with the more obvious
for start fixnum = 1 then (1+ end)
and end = (position-if … )
… thus the slightly less beautiful SetQ
This is, to my eyes, easier to follow than a complex (Do … ) form …
More for completeness than usefulness
fixup, read-char in #* reader “macro”
EMPTYP is not in :CL
comments
Clarify meaning of error message
also put nice warning icons
list the names in the CError
add format # option
I honestly had no idea this existed, and now feel foolish at having done
apparently unnecessary things more than once to have the same effect.
“In place of a prefix parameter to a directive, … You may also use the
character # in place of a parameter; it represents the number of
arguments remaining to be processed.”
use top-level eval in hosted mode?
Duplicate T/NIL?
Ensure *ROOT* is bound
search for types in global environment also
reformat
reindent
Move symbols to JSCL/CL package
Fix some compiler warnings in SBCL
Note, DO-SEQUENCE signals warnings once the arg has already been
inferred to be a list or array (by the etypecase) because it generates
code for both cases, so one branch is known to always be never taken —
the macro could be improved, but just using DOLIST and DOTIMES directly
was easier.
Fix up JSCL/CL symbols
emulate lisp-to-js (no-op)
emulate OSET as (SETF OGET)
Move HASH-TABLE to JSCL/CL; use storage-vectors
The storage-vector implementation here is really just to take advantage
of the KIND field, and still stuffs everything into a JS Object; ie,
this is a (VECTOR 1 JS:Object) with a KIND property attached. To make
the KIND more meaningful, it's stored always as a symbol (not a function
directly); ie, (make-hash-table :test #'eql) is converted backwards to
yield a kind of (HASH-TABLE EQL) — the symbol (QUOTE EQL) not the
function-object (FUNCTION EQL).
EQUALP is still a FIXME/TODO, for now.
Export some MOP symbols
fixup typo
fix up symbol packages
fix typecase bug in duplicate detection
TYPECASE does not imply (OR…)
I don't know why I had a bout of Stupid when I wrote this, but TYPECASE
isn't like CASE; if you want (OR TYPE1 TYPE2) you have to spell it out.
Do-All-External-Symbols is not in CL
Move to JSCL/CL; clean DESCRIBE
Take advantage of nascient type system and honour the STREAM parameter;
also use ~vT and *Print-Right-Margin* for formatting.
Output of DESCRIBE changed a bit due to these changes.
NB: TODO: The REPL should set *Print-Right-Margin* the way Slime does,
probably hooking into window resize events or something.
Clean up some utility functions
Initial DEFCLASS sketch
reformat
remove split-up file
reformat
reformat
begin CLOSsifying conditions
fixups for class integration
methods as classes
multiple initarg aliases
reformat
format ~:[ and ~@[ partial
finding COPYING file
CONDITION-P, fixes
namespace FSet
run-command incovation
host name, not domain name
JSCL/JS::FSET emulation
warnings
re-order in dependency order
log of rebuild errors
remove log stash
fixup of “real” ERROR
Leave the definition in boot.lisp for early in the bootstrap process, only
ERROR coöperate with conditions.lisp
Don't revert to the poorer definition, in case the better one in
conditions.lisp may have been handled already.
fix call to REDUCE
EQL-Specializer-p is in MOP package
ignore log files
correct comments
also unused field
format
remove unwanted macro redefinition
legacy of before moving to JSCL/CL package
unnecessary package prefices
re-organize ordering
move function
define all standard classes
Note that this introduces a FIXME on the class of these types.
don't stop if it didn't fail
fixes to reporting warnings/failures
fix docstring
get-internal-real-time
log builds
always report warnings/failures
fixups to type-of
Breaking out bootstrap package also
comments
ctags
eliminate compiler warnings
ambiguity in `Do-Source'
over-optimized loop in `Git-Credits'
Export bootstrapping from JSCL/Bootstrap package
fix comment
ensure the first defpackage gets evaluated at load-time
missing In-Package
moved fn package
fix up package renames
package renames
package renamed fixups
renaming packages for second pass: fix
label file
compile-file: returning three values
ignore condition object
spurious space
load FASLs only on host pass
clarify messages with in which pass errors occur
not-yet-working cross-compilation (second pass)
symbols not yet defined
Use "intern" so code loads
jscl:test-files not yet defined
Use (primitive) IF before macro UNLESS
with-compilation-environment
fixups to README
Don't allow JS reserved words as identifiers
The rather long list of reserved words in JavaScript should never be
used for identifiers. That blacklist, from
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar
is now incorporated into Valid-JS-Identifier.
valid-js-identifier-p (add -p)
Fix up bootstrap package name in Makefile
Export Load-JSCL
fix whitespace errors
Try to recover from aborted loads
APPLY constructor to args (not FUNCALL)
ensure creation of KEYWORD package (needed by Reader)
fix emulation of JS Symbol ctor
SBCL, at least, doesn't support :if-exists :new-version
This is probably true for all Unix-based Lisps, since the tradition is
to use ~n~ or a VCS on Unix/POSIX/Linux systems, rather than actual file
versions; but WebDAV *does* versioning, so JSCL should support it once
the DAV-based filesystem (at least) is in place.
fixup unwind-protect forms around second-stage cross-compilation
fix indentation
better recovery from failures in Load-JSCL
give up on deleting the current package
change JSCL/JS::Var macro
remove unused var
rename “this” as “*This*”
package designator is a string designator
package creation in bootstrap
%make-package work
package of JSCL/JS symbols
reformat
JSCL/Impl package
No second-pass package renaming
clean up package refs
verbose messaging
unnecessary distinction
fix up boot.lisp package creation
DESCRIBE: ignore errors for non-type/class; mention special forms
Trying (failing) to ensure Special Forms are processed
The CONVERT-* family of functions are not catching special forms like
LET and ∴ the compilation crashes by incorrectly believing them to be
function calls, and trying to evaluate the variable binding forms as
function calls. EG:
(let ((a b)) a) → (funcall ((a b)) a) → error, (a b) is not a fn designator
Ensure CL symbols are exported in both hosted JSCL and host CL
hash-table EQ map symbols twice
fix assertions in ~:[ ~; ~]
fix name interning for define-compilation
teletype-p for ECL
Thanks @jackdaniel
ECL stream-clear-output
special forms
the special-forms/macros bug
crashing
special-forms
storage-vector literals
merge special-operator-p
fix assert format string
qualified name on type in FTYPE
initargs (plural)
reading structures
return the storage-vector
fix Literal-SV
missing defvar (in build order)
fix messages
change order so pathnames are caught sooner
reorder
wip
|
Done |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Currently
typecase,etypecaseandtypepwork for a few hardcoded cases. No more general types are supported.It is possible to start writing a real implementation of the type system, whose main entry point is
typep.subtypepwould be also useful as it can be a basis for some compiler optimizations.The text was updated successfully, but these errors were encountered: