forked from jscl-project/jscl
-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
vector-push-extend should honor fill-pointer #52
Comments
brpocock
added a commit
that referenced
this issue
Dec 7, 2016
> This is a very large flattening effort. This branch's goal is to > stabilize the many changes on this fork; > > 1. So that the downstream use, in Tootsville, will be able to proceed > with these advantages. > > 2. So that patches from this can be made and submitted for > upstream (JSCL-Project)'s consideration and discussion. > Hopefully many of these may ultimately be adopted upstream, so the > forks will not be so very different. > > It is not stable yet, and you should see the very end that there are > issues resolved and not. There are also many TODO / HACK /FIXME > comments, some personal (@BRFennPocock) notes tagged with ☠, and the > most noticeable, it currently doesn't even compile without crashing. > > Here are the changes, roughly grouped in a kind of sensible way, > I hope. These are abridged from many commit logs and might repeat > themselves some times. User (Compiler-running programmer) Æsthetics • Improve / add some warning messages and safety checks. Ensure that JavaScript warnings/errors will have a stack trace associated with them by throwing an |Error| object. • Some errors in JavaScript are more verbose/detailed. • Non-numeric arguments are complained-about in great detail. • out-of-range AREF on a storage vector as well. • Don't moan about unbounds in SBCL CL Yes, there are symbols defined in Common-Lisp which are neither BoundP nor FBoundP, but it's not interesting to report those in SBCL. When JSCL is the CL package, there are many more “interesting” unbounds, so we want to know about those. • Generated names in both JavaScript and Lisp code more often (almost always) use GenSym with “interesting” prefixes (and similar) to make debugging a little more intuitive. eg (let ((new-value (gensym "NEW-VALUE-"))) …) • Most messages and things use the CL tradition of leading with ~& rather than trailing with ~%. Where this was inconsistent, it was changed. • Check Situations in Eval-When for typos. • Warnings about improper behavior known to exist in DefPackage and In-Package (although these bugs may have been eliminated) • Many SBCL compile-time warnings resolved. Pulled from upstream • ANSI LOOP implementation. Some of the work done upstream to fix things for JSCL are actually “un-done” now, as the type system work, changes to macroexpansion, and other ANSI-CL-fixes made them no longer necessary. Still, I did remove some reader conditionals that might have broken the LOOP code for some pre-1990-ish compilers, so the code was not exactly returned to its pristine MIT/Symbolics implementation. • storage-vector DefStruct work — this actually was quite similar to, and merged fairly transparently with, work on this branch already. • @davazp: Add ignored size argument to make-hash-table • @davazp: Replace our mapping functions by SBCL's, so we have mapcon and mapcan for free too. Implement function NReverse. • Implement multiple-value-setq (@davazp as well I think) • @davazp had introduced his own JSIZE-SYMBOL and such, but still using the SAFE-JS-FUN-NAME in preference to new JSIZE function since they serve approximately the same purpose, but the SAFE-JS-* functions are used throughout this branch already. This will need to be harmonized at some point though. • cleanup reading invalid symbols — also @davazp or else I forget what I had meant by this. • Any number of smaller commits also Unnecessary(?) changes • DO/DO* compile from a common core • All files auto-formatted by Emacs with JSCL loaded (so macro-definitions would be known). This is rude to impose upon anyone else, and I can try to conform to individual files upstream when submitting patches, but I use the autoformatter so that I don't have to think/worry about formatting different files in different ways. It also makes my own branch merges more pleasant. • I use page breaks to navigate in Emacs and so my code is often littered with them. ANSI Common Lisp compliance • DefPackage: Handle nicknames, exports (not yet documentation, but the “manual” generator runs in SBCL) • A very poor implementation of Redefine-Package • Symbol names with too many :'s should signal a reader error. • The host's Read-Char/Peek-Char are used to enable real Gray streams on the host. JSCL's Read-Char/Peek-Char are corrected to match ANSI signatures. • LOOP (as above) — some changes here to integrate with other things as well. • Not ANSI but CLtL2: DECLARATION-INFORMATION, wanted by LOOP. Presently hard-coded to claim: (optimize '((speed 1) (debug 1) (space 1) (safety 2) (compilation-speed 1)) • FORMAT options • ~:R for ordinal numbers in English. • ~R for cardinal numbers is not really written; there exists a stub. • ~@r for Roman numerals, up to 399,999. I use the Unicode versions because they display more nicely (usually). I guess a switch to use the ASCII characters like SBCL might be useful to someone else. The highest Roman numeral defined in Unicode is ↈ (“Roman Numeral One Hundred Thousand”). Displaying a larger number is not possible. The ~:@r is interpreted the same but replaces Ⅳ with ⅠⅠⅠⅠ. It does not, however, change eg. ⅩⅬ to ⅩⅩⅩⅩ or so forth. (The only use I know of this quirk is to display time of day.) • ~( ~) should handle all four case-changing modes. • ~#\Newline should work as advertised and print nothing, not even leading spaces on the subsequent line. • ~@p and ~@:P for “-y” “-ies” should now work. • ~| → #\Page • FORMAT handles (at least simple) ~{ ~} lists. Nested ~{ ~} groups should work as well. • FORMAT ~/funcall/ • ~; and ~^ are recognized for grouping. ~{~^~} works; ~; will be used by ~[~] and ~<~> which are not yet implemented. • Define JSCL/MOP and JSCL/CLTL2 for those functions that don't belong in the Common-Lisp package. • Whitespace includes Linefeed, Return, Space, Tab, Page (Form Feed), and Vertical (Line) Tab. • Name most GenSym and GemSym-like objects so that they can be more easily traced out. • “Type System” implementation (qv, below) • Significant “standardization” of the READer to use CL functions rather than private versions (eg, PEEK-CHAR vs %PEEK-CHAR) to move toward Gray streams (or similar) • quasi-quoted (back-quoted) comma-dot should now work, but I have no practical examples to test the edge cases between comma-at and comma-dot: `(x ,. y) vs `(x ,@ y) • Cross-compilation Quasiquote handling When cross-compiling, the host Lisp (SBCL)'s reader can leave quasiquotes (backquotes) in its own internal form. This defines the backquote as though it were a macro, and binds it to an acceptable (but not public) SB-IMPL::EXPAND-QUASIQUOTE to handle rewriting the form. There's a possibility of some weirdness with this, but cross-compilation tends to be weird some days. • Treat pathnames as self-evaluating literals. Some basis for (eventually) using PATHNAME objects for URI's may have made it in. • use PARSE-BODY in macroexpander properly • FDefinition, FBoundP, FMakUnbound • Slightly safer COMMON-LISP exports list General Compiler work • Bad JavaScript detection There were cases — I think these were all fixed in the “Unicode and Radices” branch — in which invalid UTF-8 sequences or #\NULL bytes could be created by the code generator, and there are now explicit checks in compile-file.lisp to detect these cases. • optimize out some pure functions The JSCL::Pure declaration on a function is meant to mean more-or-less what a Haskell programmer or mathematician would expect: The function's outputs are precisely and always determined completely by its inputs, and there are no side-effects, including object construction. EG: arithmetic. This does primitive compile-time folding of pure functions. It should be at least smart enough to collapse (+ 2 2) → 4, and with some tagging work (floor 4 2) → (values 2 0) as well. This needs more work to be more capable, but it's a start. • Proclamation code refactored a little and PURE added for functions. • Define AND, OR, WHEN, UNLESS in terms of COND, and make COND somewhat more aggressive at removing unreachable code. COND will identify unreachable code, eliminate it, and issue a warning (during cross-compilation: a continuable error) about the unreachable cases. COND is already used for [E]CASE, so they obtain that benefit as well. These optimizations could have been done in each place individually, but it seemed to make sense to use the most general — COND — branching construct to define the others. IF is a compiler primitive, so it does not yet gain this benefit. JSCL JavaScript FFI & JS codegen • Character output to the console log in browser is buffered, and flushed by #\Return #\Newline or #\Page, or writing a subsequent string. • Define JSCL/FFI package for OGet and friends. The heart of this is to provide a useful JSCL/FFI package and support building actually usable web code from an independent build from JSCL itself. The real “test case” is in the Violet Volts mesh networking code, but there should be sufficient unit tests here to gain confidence. In the furtherance of that, some CLHS compatibility features have been added, and certain JSCL features may have been refactored into smaller pieces for either testing/debugging convenience, to enable re-use, or sometimes for æsthetics (meaning that I had trouble understanding the code until I broke it down). Some of these “fixes” are SBCL-specific. • Some fixes to prelude.js to ensure “public” JS symbols are not renamed during minification. • Magic glue for passing complex objects across the FFI boundary and remembering what Lisp type we would like them to be when they come back. Most of this code has not landed, but the idea is that when a Lisp object is converted to a JavaScript one, adding a member to that object with the extremely-unlikely-to-be-altered name of #\u+1f3db “Classical Building” (arbitrary but slightly punny choice) with the string name of the Lisp class to which we would like it converted back if it is returned to us. This does not help with objects newly constructed by JavaScript code, but it could be helpful when we allow JS to mutate a Lisp object. This code is pretty much experimental and not yet very useful. • “package” is a JS keyword (ES6?) — fix error in prelude.js that upset minifiers • use new JSCL/FFI for #j: reader • #J: reader supported in host for cross-compile — although mostly useless, it won't signal errors. • writing rationals out as ratios: so e.g. 1/3 in Lisp — ie, rational 1/3 — can be passed to JavaScript as (1/3) — ie, ( 1 ÷ 3 ) ≈ .3333333… There is some support for checking whether a rational can be expressed exactly in a float, and it doesn't actually do any fancy “denominator is a valid power of two” work, it just compares them, so there's still a possible slight rounding error. But, at least, don't complain about 1/2 being rounded to a float, because it's mostly harmless. • avoid writing Lisp floats in such a way that JavaScript can't interpret them (eg, 1.0d0) • pathname conversion to JS (poorly) The goal I have is for pathname-host to be the protocol + host + port for network streams, so #p"https://example.com:123/foo/bar.lisp" = (make-pathname :host "https://example.com:123" :directory '(:absolute "foo") :name "bar" :type "lisp" :version :newest) — this means that pathname-host matches the same-origin-policy rules. NB WebDAV has pathname-version support. Build System • Exports JSCL::Compile-Application • Some tweaks to .gitignore • Cosmetic: repl-node.js is made executable and named jscl-repl instead, to “look like” the more usual command-line program (which it is). • Some (small) amount of support for compiling with non-SBCL hosts is in place. It is mostly useless still, but it is a little effort in that direction. Stubs in the Makefile allow you to watch it fail with “LISP=$whatever make” • Merge fixes to enable building JS from within another Lisp program from SBCL (incomplete) • Consistently use in *FEATURES* :JSCL-XC for the cross-compilation of JSCL itself, and :JSCL for only when JSCL is the runtime system. IE: By examining *FEATURES*, determine whether JSCL provides Common-Lisp. • add a JSCL:Bootstrap-Core that does not build the REPLs • Replace make.sh and run-tests with a simple Makefile Upstream might not like this, but it works more nicely as a git submodule within another project • I had experimented with compile-time progress reporting and decided it was too much hassle running in different non-TTY environments (Slime, Travis CI, etc) and ripped it back out. The compile process is a bit more verbose and compiler messages now look a bit more like, eg, SBCL's — they make up a hierarchy of comments with more semicolons for higher-level messages. • Build REPLs in their own packages as well. • set *COMPILE-FILE-PATHNAME* always • Emit-Uncompilable-Form — One of the somewhat nice things about Java is that an uncompilable form can be emitted as a runtime error, in the hopes that nobody will trip over your horrible mistakes. This gives you the option of trying to proceed with compilation when things have gone wrong — it's unlikely to be a good thing in the long run, but it can sometimes be useful during rapid development. • verbose build banner Mostly just so I can keep track of where it restarted in the REPL when I'm iterating over things. • disable SBCL's debugger by default during build (for CI reasons); you can re-enable it with “LISPFLAGS= make” (blank after LISPFLAGS) • compile-file.lisp broken out from jscl.lisp and refactored into smaller pieces. May conform a little better to ANSI CL behavior also. • Tag generated files with compiler version and source files until we get Source Maps. This includes the Git revision level and there is a flag meant to be set on our branch to distinguish it from upstream. • Uniformly warn about target-only files (signal an error if they're loaded with #-JSCL-XC) This keeps me from accidentally making SBCL angry during development. It is such an instinct to recompile after editing … • When running in the host environment, if a macro-function is defined in the (host's) Common-Lisp package, and no matching macro-function is found in JSCL's, then a !-prefixed version will be used instead. This follows from jscl-project#35 but I think there is a nicer solution to replace it in future, using separate packages. This is in Check-for-Failed-MacroExpansion. Documentation • Comments in many cases were changed instead to docstrings. Some elucidations added. • Some other docstrings were added; in a couple of cases, from other compilers, particularly SBCL. • Info, PDF and HTML versions of the docstrings should now be generated by way of DECLT. Requires Quicklisp on the host compiler and Info. This is rather rudimentary still. While PDF and HTML are nice, integrating the Info with Emacs and Slime is a key goal in future. Integrating the HTML version with DESCRIBE in the Web REPL might be a fun thing also. XXX • The DECLT in Quicklisp crashed in SBCL for me, a hack to make it work is embedded in write-docs.lisp, I need to see if this should go upstream or if I've broken something locally. TODO • Very silly stub files for the manual exist for introduction, conclusion, and HTML output. • An ASDF file exists, only to show JSCL where to look for the files. It has a cloned copy of extract-version-from-package.json also. I guess we may the the first CL program to really want to write a package.json as a result of an ASDF build some day. Tests • Some tests that should work now are un-commented. • Don't run FFI tests in SBCL • Report package of unbound exports • skip (cond (1)) test This is probably a bad one to skip, but for right now may be forgiveable. FIXME • Travis CI & Circle CI YAML tweaks • The built-in “test reporting” code is significantly more verbose and has a few changes that were meant to be useful for seeking ANSI compliance. In particular, failed tests are gathered and reported at the end, and symbols that SBCL binds in the Common-Lisp package but are absent from JSCL are reported as well. These changes sacrificed, for the moment, the ability to generate HTML output correctly. This should probably be cured before offering these patches upstream. TODO • The GCL ANSI test suite is now included as a submodule, but there is not any code to actually invoke it. • The new TEST-FN from upstream was pasted into the file but not used, because it will require “rewiring” some things. TODO SetF, &Environment, and related work • Support for SetF functions in DEFUN and FDefinition and so forth. Basically, the big change that happened some time between CLtL1 and ANSI CL to allow (defun (setf foo) (new-value …) …) also nominally allows the implementation to allow (defun (X Y) (…) …). So, this batch of changes covers a few things. Firstly, FDefinition and DEFUN and Apply and FunCall (and the compiler implicit function application) should all allow a Function-Name-P form rather than only a symbol. Secondly, the process of finding whether a SetF expander function or a function bound to (SetF X) is used had to be reworked a bit. Also, I introduced some hooks that should be consistent with allowing JSCL/FFI:OGet forms in the Function-Name-P space, as a JSCL extension that I believe complies with the standard's requirements. These are not all fully implementated, but it aligns with using the #J reader macro to expand an OGet form in the CAR position, ie, (lambda (x) (#j:foo x)) → (lambda (x) ((jscl/ffi:oget *root* "foo") x) ✓ In the FSet context, this extends the compiler primitives and prelude.js to add FSet-SetF and give symbols in the package space a SetF-value; so the function (setf foo) appears as the SetF value of the symbol (intern "FOO"). I have some concerns about the integration of the FSet-type function definitions (package-interned symbols) and the Environment-type function definitions which also apply to SetFs; see the notes on those commits. This could also be extended to support ((macro-function foo) whole &optional (environment *environment*)) as SBCL more-or-less does, if that interests anyone. • (defun (setf foo) … ) defines a (block foo … ) for eg Return-From. • setf ƒ for car, cdr, nth • Support for the Nil or Global environment directly via macro-expansions. Lambda-list support for &Whole and &Environment variables. Macro-Functions should now all get the &whole and &environment parameters, where the environment defaults to jscl::*environment* when not supplied, but a new jscl::*global-environment* exists which can be accessed (as sometimes required by ANSI CL) by passing an explicit NIL environment, eg, for top-level forms. Notably, DEFUN is defined to affect the NIL environment if it is considered a top-level form, which is usually the case (even in a top-level ProgN or so forth). eg, (let ((x 42)) (defun y () (incf x))) — must bind x in the LexEnv *environment* bound by LET, but Y in the NIL (global) environment, with a reference back to X in the LexEnv. *Global-Environment* allows DEFUN to “reach up” and work this way, and a simple hack in the compiler falls back to searching the global environment when *environment* does not have a definition. This does not handle the case in which a symbol was globally redefined between the time at which *environment* was cloned from the global environment and the time at which a form in that LexEnv was compiled, which is an edge case but almost certainly a BUG. There is a conflict of representation here between the LexEnv structures used by *Environment* and the low-level FSet-type bindings in prelude.js — the resolution to this may be to remove the prelude/FSet-type code perhaps, and define packages entirely in terms of the *Global-Environment*? I haven't explored this very thoroughly, but it seems possible for the global environment to fall out-of-sync with the JavaScript definitions, and bits of the code to look in one place or the other improperly. An alternative might be to not use *global-environment* at all, but to push all global definitions up to the JavaScript symbols (for every kind of slot on the LexEnv structure). This sounds nicer, but will probably require a little more “strange” changes to the compiler-level code, and for cross-compilation, require emulation in compat.lisp to keep track of these changes. (It probably also means some interesting tables to be emitted in the “fasl” JavaScript files to populate these tables, but that is the point of the FSet declarations that now exist anyway.) • Don't allow (fdefinition '(function foo)) Pretty sure that's not a good thing. SBCL signals an error and the CLHS defines function-name as > A symbol or a list (setf symbol). We extend that for FFI with list (oget …) also, but (function symbol) isn't really a thing … It's kinda recursive, really, since you'd expect (function symbol) to be essentially a macro version of (symbol-function (quote symbol)) which in turn is (fdefinition (quote symbol)) … I've talked myself into destroying it. Type System These changes are pretty far-reaching, invasive, and more importantly 👿 👺 THIS CODE IS STILL BROKEN 🔥 👹 • Type system core: • Hierarchy of types for those types we actually support currently • Type definitions for built-in types • Compound type definitions for strings, arrays, and numeric types • DefType understanding basic type names, SATISFIES, MOD, AND, OR, MEMBER, and NOT. Does not work right with EQL or compound types like (INTEGER 0 *) yet. (VALUES) types are not handled either. • Type-Of. Returns the type name symbol, but can return a compound type form describing precisely a string, array, or integer; eg, (STRING CHARACTER 42) for a string of 42 characters, or (ARRAY T (2 3)), or (INTEGER 9 9). • Class-Of • Reworked TypeP to use the above, mostly. • Reworked TypeCase to use the new TypeP • Reworked ETypeCase slightly • Supertypes are all defined as in ANSI, but, many of the types named do not actually exist: particularly streams and conditions, but also some variant numeric types (different kinds of floats, complex, rational) and Simple-Base-String and so forth. • Some support for EQL types for DefType and (later) DefGeneric/DefMethod. • Lexical symbols can have bound to them setf-ƒ, and can name a type or a class. (They do not yet have property lists, or documentation of various types.) • “Impassible” cases in TypeCase and ETypeCase called out at compile-time. EG: (typecase 42 (number 'num) (integer 'int)) should signal a warning. Curiously, SBCL doesn't always notice the unreachable code in these cases. • Duplicated (exactly) type names issue a warning also • Numeric types that overlap in ranges (eg, (INTEGER 0 4) and (MOD 5)) are NOT yet caught. • Merge upstream branch changes to DefStruct options and structures in storage vectors from @davazp — I'm afraid the actual commits probably got merged down in the squash process along with my own. • Storage-Vectors are used for arrays, structure objects, and (eventually) other kinds of objects. The “kind” of thing is now stored alongside the underlying vector. I avoid the words “type” or “class” here to distinguish that this is a magic cookie CONS that describes what may-or-may-not be an instance of a “type” or “class” in either of Lisp or JavaScript. The definition of the “kind” is that it is a list, the FIRST of which is a general type, and the REST of which is defined in terms of that general type. For an array, the dimensions are stored here. For a structure object, the structure type is given (the layout is only known by looking up that symbol as a type). For object instances, the idea is to have a pointer to the metaobject layout (allowing layout upgrading) placed here. For now, the FIRST of KIND is of type (MEMBER ARRAY STANDARD-OBJECT STRUCTURE-OBJECT). Metaobject layouts and generic function objects might extend that list. • TypeP 'FixNumP only for numbers in the integer range of JavaScript: ±(2⁵³-1) • Handle isolation of the cases of SBCL or JSCL dealing with SBCL or JSCL-targetted code, package namespaces, and things like that. • leave early-char code in early-char, remove “incorrect” character code stuff from boot.lisp • define TypeCase in terms of TypeP • Some forward declarations between defstruct.lisp and types.lisp. This is decidedly bad “code smell” and should theoretically not be actually needed in the long run, but it does help to quash some problems for now. There is a pattern emerging that suggests that certain bits of types.lisp should be split into two modules, loaded before and after defstruct.lisp … —~— Remaining Open Issues that might not be merged into this branch. Some of these may be trivial to cure on this branch, and could “make it in.” #66 #68 #62 #53 #52 #51 #49 #47 #36 #40 #33 jscl-project#261 jscl-project#243 jscl-project#233 jscl-project#203 jscl-project#176 jscl-project#159 jscl-project#127 To be closed with this branch, but not closing Issues tickets until this branch can build and pass tests. The upstream issues of course will also need to be parcelled out into more manageable patches. #67 #64 # partial #63 #50 #48 #45 #44 #34 #30 jscl-project#259 jscl-project#258 jscl-project#228 jscl-project#192 # partial? jscl-project#164 # partial jscl-project#17 # partial —~ 𝒻𝒾𝓃𝒾𝓈 ~— fix negative hex When radix is being printed, it should appear before the sign ⇒ (let ((*print-radix* t) (*print-base* 16)) (format t "~a" -42)) #x-2A additional type check Grab the GCL ANSI tests Random puttering Still having issues with not-expanded macros and also with SBCL not liking structure-objects that were defined in the JSCL (storage-vector) way. Keep patch to ANSI-Test here. The current patch is from @JackDaniel and will probably make it into the Git version soon enough, but for the moment, I don't expect there to be “enough” things that we need to patch in the test suite itself to want to maintain a fork. More significantly, for things that are so badly broken that we have to patch the test suite (eg: perhaps a reader macro bug or something), the thing to do is to fix JSCL; in the mean time, this patch file can live as a “pebble in the sandal” so that disabled, broken tests are never forgotten completely. Ignores: renamed targets, re-organize file build docs Stabilization, debugging. Trying to get a clean build from which to close out the remaining critical issues and begin parcelling out patches to upstream. Includes some build engineering work and documentation fixes as well as tracking down build failures. two horribly stupid typos documentation mop up various compile-time errors types, macros, and format options Work on the new Type System infrastructure, and normalizing the code between hosted and self-hosting operations Some more work to get macro-expansion at compile-time working properly for cross-compilation Additional format handlers — `~r`, `~:r`, `~@r`, beginning to work on `~{ ~^ ~}` and `~[ ~]` and friends. sketch in support for other hosts ECL support more SBCL-independence fixups; quasi-quote changes did not work Ignore Emacs project-type files FORMAT ~/funcall/ operation Don't error out on forward-declared functions. This seems to be a bit more complicated than necessary, but it has to do with the awful circularity between the type system and the compiler itself. Perhaps if the low-level bindings are factored out into a quite early module, some of this can be eliminated. FSet-SetF does not neet early Eval-When Share JavaScript's MOST-{POSITIVE,NEGATIVE}-FIXNUM The compiler needs to be on the lookout for bignums, so these figures are duplicated at +MOST-{POSITIVE,NEGATIVE}-FIXNUM+ to be visible in the host compiler as well. confusion over (FUNCTION NAME) This was probably more of a problem with formatting the error message, than with the code, but to avoid the mystifying “#’NAME” messages, renamed the parameter here to FN Forward-declare functions & vars from compiler Types fixups: building hierarchy, Typecases Move all the definitions of built-in-types into a single function to establish the types and classes required by the standard (note: many/most of these, particularly conditions and CLOS things, are not actually defined to have any behavior or even be possible to make instances of, but the type names at least will be known to have the proper subtype relationships) Also trying to clean up some horrible error behaviour from [E]TypeCase, which is not yet solved at this commit Misc compiler fixes: - Fix some SetF definitions - Support typing attributes for Storage-Vectors - Move DECLARATION-INFORMATION to JSCL/CLTL2 - Be more conservative about literals that do not (yet) work in JS - Warn about bad Eval-When situations Add a no-op WITH-COMPILATION-UNIT warn unreachable subtypes in TYPECASE *** empty log message *** make declaration match the actual inferred type later Hosted macroexpansion is the bitch The semi-circular-reference hack Æsthetics of compiling DEFTYPE should EVAL BODY but this nasty hack works for a temp patch dumping function pointers … trashy hack Fix JSCL banner (violet-volts-p) out of order
brpocock
added a commit
that referenced
this issue
Mar 25, 2017
warnings and such DEFPACKAGE :EXPORT handler define FFI package. Need to merge in from other branch. (defpackage (:export …)) Interpret page break character as whitespace Implement copier constructor and predicate optiosn for def!struct Add a fake fixnum case as an integer Add ignored size argument to make-hash-table Replace our mapping functions by SBCL's So we have mapcon and mapcan for free too. Implement function NREVERSE Define dummy subtypep function Implement multiple-value-setq Provide dummy CONSTANTP function Define the 'the' special form For now, it does nothing Expose macroexpand-1 function Improve stacktraces by naming lambda functions in JS macroexpand-1 accepts optional environment argument Provide !macroexpand function Fix defun The jsize-symbol function needs to generate a valid JS identifier, otherwise the code generator will fail. So we have to be careful so we do not print the symbols with the package prefix. Rebase from upstream master Also, using the SAFE-JS-FUN-NAME in preference to new JSIZE function since they serve approximately the same purpose, but the SAFE-JS-* functions are used throughout this branch already. Tests for Unicode, numeric radix Note that these will usually crash the master branch. merge conflicts fix Terribly stupid git merge markers made it in. add forgotten FORMAT ~b, ~o Fix FORMAT padding. Add docstrings. uncomment now-working test; add a couple easy other tests mass reformat; missing ) (a couple of close-parens got lost in the merge) Rolling up JSCL changes — Stabilization > This is a very large flattening effort. This branch's goal is to > stabilize the many changes on this fork; > > 1. So that the downstream use, in Tootsville, will be able to proceed > with these advantages. > > 2. So that patches from this can be made and submitted for > upstream (JSCL-Project)'s consideration and discussion. > Hopefully many of these may ultimately be adopted upstream, so the > forks will not be so very different. > > It is not stable yet, and you should see the very end that there are > issues resolved and not. There are also many TODO / HACK /FIXME > comments, some personal (@BRFennPocock) notes tagged with ☠, and the > most noticeable, it currently doesn't even compile without crashing. > > Here are the changes, roughly grouped in a kind of sensible way, > I hope. These are abridged from many commit logs and might repeat > themselves some times. User (Compiler-running programmer) Æsthetics • Improve / add some warning messages and safety checks. Ensure that JavaScript warnings/errors will have a stack trace associated with them by throwing an |Error| object. • Some errors in JavaScript are more verbose/detailed. • Non-numeric arguments are complained-about in great detail. • out-of-range AREF on a storage vector as well. • Don't moan about unbounds in SBCL CL Yes, there are symbols defined in Common-Lisp which are neither BoundP nor FBoundP, but it's not interesting to report those in SBCL. When JSCL is the CL package, there are many more “interesting” unbounds, so we want to know about those. • Generated names in both JavaScript and Lisp code more often (almost always) use GenSym with “interesting” prefixes (and similar) to make debugging a little more intuitive. eg (let ((new-value (gensym "NEW-VALUE-"))) …) • Most messages and things use the CL tradition of leading with ~& rather than trailing with ~%. Where this was inconsistent, it was changed. • Check Situations in Eval-When for typos. • Warnings about improper behavior known to exist in DefPackage and In-Package (although these bugs may have been eliminated) • Many SBCL compile-time warnings resolved. Pulled from upstream • ANSI LOOP implementation. Some of the work done upstream to fix things for JSCL are actually “un-done” now, as the type system work, changes to macroexpansion, and other ANSI-CL-fixes made them no longer necessary. Still, I did remove some reader conditionals that might have broken the LOOP code for some pre-1990-ish compilers, so the code was not exactly returned to its pristine MIT/Symbolics implementation. • storage-vector DefStruct work — this actually was quite similar to, and merged fairly transparently with, work on this branch already. • @davazp: Add ignored size argument to make-hash-table • @davazp: Replace our mapping functions by SBCL's, so we have mapcon and mapcan for free too. Implement function NReverse. • Implement multiple-value-setq (@davazp as well I think) • @davazp had introduced his own JSIZE-SYMBOL and such, but still using the SAFE-JS-FUN-NAME in preference to new JSIZE function since they serve approximately the same purpose, but the SAFE-JS-* functions are used throughout this branch already. This will need to be harmonized at some point though. • cleanup reading invalid symbols — also @davazp or else I forget what I had meant by this. • Any number of smaller commits also Unnecessary(?) changes • DO/DO* compile from a common core • All files auto-formatted by Emacs with JSCL loaded (so macro-definitions would be known). This is rude to impose upon anyone else, and I can try to conform to individual files upstream when submitting patches, but I use the autoformatter so that I don't have to think/worry about formatting different files in different ways. It also makes my own branch merges more pleasant. • I use page breaks to navigate in Emacs and so my code is often littered with them. ANSI Common Lisp compliance • DefPackage: Handle nicknames, exports (not yet documentation, but the “manual” generator runs in SBCL) • A very poor implementation of Redefine-Package • Symbol names with too many :'s should signal a reader error. • The host's Read-Char/Peek-Char are used to enable real Gray streams on the host. JSCL's Read-Char/Peek-Char are corrected to match ANSI signatures. • LOOP (as above) — some changes here to integrate with other things as well. • Not ANSI but CLtL2: DECLARATION-INFORMATION, wanted by LOOP. Presently hard-coded to claim: (optimize '((speed 1) (debug 1) (space 1) (safety 2) (compilation-speed 1)) • FORMAT options • ~:R for ordinal numbers in English. • ~R for cardinal numbers is not really written; there exists a stub. • ~@R for Roman numerals, up to 399,999. I use the Unicode versions because they display more nicely (usually). I guess a switch to use the ASCII characters like SBCL might be useful to someone else. The highest Roman numeral defined in Unicode is ↈ (“Roman Numeral One Hundred Thousand”). Displaying a larger number is not possible. The ~:@R is interpreted the same but replaces Ⅳ with ⅠⅠⅠⅠ. It does not, however, change eg. ⅩⅬ to ⅩⅩⅩⅩ or so forth. (The only use I know of this quirk is to display time of day.) • ~( ~) should handle all four case-changing modes. • ~#\Newline should work as advertised and print nothing, not even leading spaces on the subsequent line. • ~@P and ~@:P for “-y” “-ies” should now work. • ~| → #\Page • FORMAT handles (at least simple) ~{ ~} lists. Nested ~{ ~} groups should work as well. • FORMAT ~/funcall/ • ~; and ~^ are recognized for grouping. ~{~^~} works; ~; will be used by ~[~] and ~<~> which are not yet implemented. • Define JSCL/MOP and JSCL/CLTL2 for those functions that don't belong in the Common-Lisp package. • Whitespace includes Linefeed, Return, Space, Tab, Page (Form Feed), and Vertical (Line) Tab. • Name most GenSym and GemSym-like objects so that they can be more easily traced out. • “Type System” implementation (qv, below) • Significant “standardization” of the READer to use CL functions rather than private versions (eg, PEEK-CHAR vs %PEEK-CHAR) to move toward Gray streams (or similar) • quasi-quoted (back-quoted) comma-dot should now work, but I have no practical examples to test the edge cases between comma-at and comma-dot: `(x ,. y) vs `(x ,@ y) • Cross-compilation Quasiquote handling When cross-compiling, the host Lisp (SBCL)'s reader can leave quasiquotes (backquotes) in its own internal form. This defines the backquote as though it were a macro, and binds it to an acceptable (but not public) SB-IMPL::EXPAND-QUASIQUOTE to handle rewriting the form. There's a possibility of some weirdness with this, but cross-compilation tends to be weird some days. • Treat pathnames as self-evaluating literals. Some basis for (eventually) using PATHNAME objects for URI's may have made it in. • use PARSE-BODY in macroexpander properly • FDefinition, FBoundP, FMakUnbound • Slightly safer COMMON-LISP exports list General Compiler work • Bad JavaScript detection There were cases — I think these were all fixed in the “Unicode and Radices” branch — in which invalid UTF-8 sequences or #\NULL bytes could be created by the code generator, and there are now explicit checks in compile-file.lisp to detect these cases. • optimize out some pure functions The JSCL::Pure declaration on a function is meant to mean more-or-less what a Haskell programmer or mathematician would expect: The function's outputs are precisely and always determined completely by its inputs, and there are no side-effects, including object construction. EG: arithmetic. This does primitive compile-time folding of pure functions. It should be at least smart enough to collapse (+ 2 2) → 4, and with some tagging work (floor 4 2) → (values 2 0) as well. This needs more work to be more capable, but it's a start. • Proclamation code refactored a little and PURE added for functions. • Define AND, OR, WHEN, UNLESS in terms of COND, and make COND somewhat more aggressive at removing unreachable code. COND will identify unreachable code, eliminate it, and issue a warning (during cross-compilation: a continuable error) about the unreachable cases. COND is already used for [E]CASE, so they obtain that benefit as well. These optimizations could have been done in each place individually, but it seemed to make sense to use the most general — COND — branching construct to define the others. IF is a compiler primitive, so it does not yet gain this benefit. JSCL JavaScript FFI & JS codegen • Character output to the console log in browser is buffered, and flushed by #\Return #\Newline or #\Page, or writing a subsequent string. • Define JSCL/FFI package for OGet and friends. The heart of this is to provide a useful JSCL/FFI package and support building actually usable web code from an independent build from JSCL itself. The real “test case” is in the Violet Volts mesh networking code, but there should be sufficient unit tests here to gain confidence. In the furtherance of that, some CLHS compatibility features have been added, and certain JSCL features may have been refactored into smaller pieces for either testing/debugging convenience, to enable re-use, or sometimes for æsthetics (meaning that I had trouble understanding the code until I broke it down). Some of these “fixes” are SBCL-specific. • Some fixes to prelude.js to ensure “public” JS symbols are not renamed during minification. • Magic glue for passing complex objects across the FFI boundary and remembering what Lisp type we would like them to be when they come back. Most of this code has not landed, but the idea is that when a Lisp object is converted to a JavaScript one, adding a member to that object with the extremely-unlikely-to-be-altered name of #\u+1f3db “Classical Building” (arbitrary but slightly punny choice) with the string name of the Lisp class to which we would like it converted back if it is returned to us. This does not help with objects newly constructed by JavaScript code, but it could be helpful when we allow JS to mutate a Lisp object. This code is pretty much experimental and not yet very useful. • “package” is a JS keyword (ES6?) — fix error in prelude.js that upset minifiers • use new JSCL/FFI for #j: reader • #J: reader supported in host for cross-compile — although mostly useless, it won't signal errors. • writing rationals out as ratios: so e.g. 1/3 in Lisp — ie, rational 1/3 — can be passed to JavaScript as (1/3) — ie, ( 1 ÷ 3 ) ≈ .3333333… There is some support for checking whether a rational can be expressed exactly in a float, and it doesn't actually do any fancy “denominator is a valid power of two” work, it just compares them, so there's still a possible slight rounding error. But, at least, don't complain about 1/2 being rounded to a float, because it's mostly harmless. • avoid writing Lisp floats in such a way that JavaScript can't interpret them (eg, 1.0d0) • pathname conversion to JS (poorly) The goal I have is for pathname-host to be the protocol + host + port for network streams, so #p"https://example.com:123/foo/bar.lisp" = (make-pathname :host "https://example.com:123" :directory '(:absolute "foo") :name "bar" :type "lisp" :version :newest) — this means that pathname-host matches the same-origin-policy rules. NB WebDAV has pathname-version support. Build System • Exports JSCL::Compile-Application • Some tweaks to .gitignore • Cosmetic: repl-node.js is made executable and named jscl-repl instead, to “look like” the more usual command-line program (which it is). • Some (small) amount of support for compiling with non-SBCL hosts is in place. It is mostly useless still, but it is a little effort in that direction. Stubs in the Makefile allow you to watch it fail with “LISP=$whatever make” • Merge fixes to enable building JS from within another Lisp program from SBCL (incomplete) • Consistently use in *FEATURES* :JSCL-XC for the cross-compilation of JSCL itself, and :JSCL for only when JSCL is the runtime system. IE: By examining *FEATURES*, determine whether JSCL provides Common-Lisp. • add a JSCL:Bootstrap-Core that does not build the REPLs • Replace make.sh and run-tests with a simple Makefile Upstream might not like this, but it works more nicely as a git submodule within another project • I had experimented with compile-time progress reporting and decided it was too much hassle running in different non-TTY environments (Slime, Travis CI, etc) and ripped it back out. The compile process is a bit more verbose and compiler messages now look a bit more like, eg, SBCL's — they make up a hierarchy of comments with more semicolons for higher-level messages. • Build REPLs in their own packages as well. • set *COMPILE-FILE-PATHNAME* always • Emit-Uncompilable-Form — One of the somewhat nice things about Java is that an uncompilable form can be emitted as a runtime error, in the hopes that nobody will trip over your horrible mistakes. This gives you the option of trying to proceed with compilation when things have gone wrong — it's unlikely to be a good thing in the long run, but it can sometimes be useful during rapid development. • verbose build banner Mostly just so I can keep track of where it restarted in the REPL when I'm iterating over things. • disable SBCL's debugger by default during build (for CI reasons); you can re-enable it with “LISPFLAGS= make” (blank after LISPFLAGS) • compile-file.lisp broken out from jscl.lisp and refactored into smaller pieces. May conform a little better to ANSI CL behavior also. • Tag generated files with compiler version and source files until we get Source Maps. This includes the Git revision level and there is a flag meant to be set on our branch to distinguish it from upstream. • Uniformly warn about target-only files (signal an error if they're loaded with #-JSCL-XC) This keeps me from accidentally making SBCL angry during development. It is such an instinct to recompile after editing … • When running in the host environment, if a macro-function is defined in the (host's) Common-Lisp package, and no matching macro-function is found in JSCL's, then a !-prefixed version will be used instead. This follows from https://github.com/jscl-project/jscl/issues/35 but I think there is a nicer solution to replace it in future, using separate packages. This is in Check-for-Failed-MacroExpansion. Documentation • Comments in many cases were changed instead to docstrings. Some elucidations added. • Some other docstrings were added; in a couple of cases, from other compilers, particularly SBCL. • Info, PDF and HTML versions of the docstrings should now be generated by way of DECLT. Requires Quicklisp on the host compiler and Info. This is rather rudimentary still. While PDF and HTML are nice, integrating the Info with Emacs and Slime is a key goal in future. Integrating the HTML version with DESCRIBE in the Web REPL might be a fun thing also. XXX • The DECLT in Quicklisp crashed in SBCL for me, a hack to make it work is embedded in write-docs.lisp, I need to see if this should go upstream or if I've broken something locally. TODO • Very silly stub files for the manual exist for introduction, conclusion, and HTML output. • An ASDF file exists, only to show JSCL where to look for the files. It has a cloned copy of extract-version-from-package.json also. I guess we may the the first CL program to really want to write a package.json as a result of an ASDF build some day. Tests • Some tests that should work now are un-commented. • Don't run FFI tests in SBCL • Report package of unbound exports • skip (cond (1)) test This is probably a bad one to skip, but for right now may be forgiveable. FIXME • Travis CI & Circle CI YAML tweaks • The built-in “test reporting” code is significantly more verbose and has a few changes that were meant to be useful for seeking ANSI compliance. In particular, failed tests are gathered and reported at the end, and symbols that SBCL binds in the Common-Lisp package but are absent from JSCL are reported as well. These changes sacrificed, for the moment, the ability to generate HTML output correctly. This should probably be cured before offering these patches upstream. TODO • The GCL ANSI test suite is now included as a submodule, but there is not any code to actually invoke it. • The new TEST-FN from upstream was pasted into the file but not used, because it will require “rewiring” some things. TODO SetF, &Environment, and related work • Support for SetF functions in DEFUN and FDefinition and so forth. Basically, the big change that happened some time between CLtL1 and ANSI CL to allow (defun (setf foo) (new-value …) …) also nominally allows the implementation to allow (defun (X Y) (…) …). So, this batch of changes covers a few things. Firstly, FDefinition and DEFUN and Apply and FunCall (and the compiler implicit function application) should all allow a Function-Name-P form rather than only a symbol. Secondly, the process of finding whether a SetF expander function or a function bound to (SetF X) is used had to be reworked a bit. Also, I introduced some hooks that should be consistent with allowing JSCL/FFI:OGet forms in the Function-Name-P space, as a JSCL extension that I believe complies with the standard's requirements. These are not all fully implementated, but it aligns with using the #J reader macro to expand an OGet form in the CAR position, ie, (lambda (x) (#j:foo x)) → (lambda (x) ((jscl/ffi:oget *root* "foo") x) ✓ In the FSet context, this extends the compiler primitives and prelude.js to add FSet-SetF and give symbols in the package space a SetF-value; so the function (setf foo) appears as the SetF value of the symbol (intern "FOO"). I have some concerns about the integration of the FSet-type function definitions (package-interned symbols) and the Environment-type function definitions which also apply to SetFs; see the notes on those commits. This could also be extended to support ((macro-function foo) whole &optional (environment *environment*)) as SBCL more-or-less does, if that interests anyone. • (defun (setf foo) … ) defines a (block foo … ) for eg Return-From. • setf ƒ for car, cdr, nth • Support for the Nil or Global environment directly via macro-expansions. Lambda-list support for &Whole and &Environment variables. Macro-Functions should now all get the &whole and &environment parameters, where the environment defaults to jscl::*environment* when not supplied, but a new jscl::*global-environment* exists which can be accessed (as sometimes required by ANSI CL) by passing an explicit NIL environment, eg, for top-level forms. Notably, DEFUN is defined to affect the NIL environment if it is considered a top-level form, which is usually the case (even in a top-level ProgN or so forth). eg, (let ((x 42)) (defun y () (incf x))) — must bind x in the LexEnv *environment* bound by LET, but Y in the NIL (global) environment, with a reference back to X in the LexEnv. *Global-Environment* allows DEFUN to “reach up” and work this way, and a simple hack in the compiler falls back to searching the global environment when *environment* does not have a definition. This does not handle the case in which a symbol was globally redefined between the time at which *environment* was cloned from the global environment and the time at which a form in that LexEnv was compiled, which is an edge case but almost certainly a BUG. There is a conflict of representation here between the LexEnv structures used by *Environment* and the low-level FSet-type bindings in prelude.js — the resolution to this may be to remove the prelude/FSet-type code perhaps, and define packages entirely in terms of the *Global-Environment*? I haven't explored this very thoroughly, but it seems possible for the global environment to fall out-of-sync with the JavaScript definitions, and bits of the code to look in one place or the other improperly. An alternative might be to not use *global-environment* at all, but to push all global definitions up to the JavaScript symbols (for every kind of slot on the LexEnv structure). This sounds nicer, but will probably require a little more “strange” changes to the compiler-level code, and for cross-compilation, require emulation in compat.lisp to keep track of these changes. (It probably also means some interesting tables to be emitted in the “fasl” JavaScript files to populate these tables, but that is the point of the FSet declarations that now exist anyway.) • Don't allow (fdefinition '(function foo)) Pretty sure that's not a good thing. SBCL signals an error and the CLHS defines function-name as > A symbol or a list (setf symbol). We extend that for FFI with list (oget …) also, but (function symbol) isn't really a thing … It's kinda recursive, really, since you'd expect (function symbol) to be essentially a macro version of (symbol-function (quote symbol)) which in turn is (fdefinition (quote symbol)) … I've talked myself into destroying it. Type System These changes are pretty far-reaching, invasive, and more importantly 👿 👺 THIS CODE IS STILL BROKEN 🔥 👹 • Type system core: • Hierarchy of types for those types we actually support currently • Type definitions for built-in types • Compound type definitions for strings, arrays, and numeric types • DefType understanding basic type names, SATISFIES, MOD, AND, OR, MEMBER, and NOT. Does not work right with EQL or compound types like (INTEGER 0 *) yet. (VALUES) types are not handled either. • Type-Of. Returns the type name symbol, but can return a compound type form describing precisely a string, array, or integer; eg, (STRING CHARACTER 42) for a string of 42 characters, or (ARRAY T (2 3)), or (INTEGER 9 9). • Class-Of • Reworked TypeP to use the above, mostly. • Reworked TypeCase to use the new TypeP • Reworked ETypeCase slightly • Supertypes are all defined as in ANSI, but, many of the types named do not actually exist: particularly streams and conditions, but also some variant numeric types (different kinds of floats, complex, rational) and Simple-Base-String and so forth. • Some support for EQL types for DefType and (later) DefGeneric/DefMethod. • Lexical symbols can have bound to them setf-ƒ, and can name a type or a class. (They do not yet have property lists, or documentation of various types.) • “Impassible” cases in TypeCase and ETypeCase called out at compile-time. EG: (typecase 42 (number 'num) (integer 'int)) should signal a warning. Curiously, SBCL doesn't always notice the unreachable code in these cases. • Duplicated (exactly) type names issue a warning also • Numeric types that overlap in ranges (eg, (INTEGER 0 4) and (MOD 5)) are NOT yet caught. • Merge upstream branch changes to DefStruct options and structures in storage vectors from @davazp — I'm afraid the actual commits probably got merged down in the squash process along with my own. • Storage-Vectors are used for arrays, structure objects, and (eventually) other kinds of objects. The “kind” of thing is now stored alongside the underlying vector. I avoid the words “type” or “class” here to distinguish that this is a magic cookie CONS that describes what may-or-may-not be an instance of a “type” or “class” in either of Lisp or JavaScript. The definition of the “kind” is that it is a list, the FIRST of which is a general type, and the REST of which is defined in terms of that general type. For an array, the dimensions are stored here. For a structure object, the structure type is given (the layout is only known by looking up that symbol as a type). For object instances, the idea is to have a pointer to the metaobject layout (allowing layout upgrading) placed here. For now, the FIRST of KIND is of type (MEMBER ARRAY STANDARD-OBJECT STRUCTURE-OBJECT). Metaobject layouts and generic function objects might extend that list. • TypeP 'FixNumP only for numbers in the integer range of JavaScript: ±(2⁵³-1) • Handle isolation of the cases of SBCL or JSCL dealing with SBCL or JSCL-targetted code, package namespaces, and things like that. • leave early-char code in early-char, remove “incorrect” character code stuff from boot.lisp • define TypeCase in terms of TypeP • Some forward declarations between defstruct.lisp and types.lisp. This is decidedly bad “code smell” and should theoretically not be actually needed in the long run, but it does help to quash some problems for now. There is a pattern emerging that suggests that certain bits of types.lisp should be split into two modules, loaded before and after defstruct.lisp … —~— Remaining Open Issues that might not be merged into this branch. Some of these may be trivial to cure on this branch, and could “make it in.” https://github.com/romance-ii/jscl/issues/66 https://github.com/romance-ii/jscl/issues/68 https://github.com/romance-ii/jscl/issues/62 https://github.com/romance-ii/jscl/issues/53 https://github.com/romance-ii/jscl/issues/52 https://github.com/romance-ii/jscl/issues/51 https://github.com/romance-ii/jscl/issues/49 https://github.com/romance-ii/jscl/issues/47 https://github.com/romance-ii/jscl/issues/36 https://github.com/romance-ii/jscl/issues/40 https://github.com/romance-ii/jscl/issues/33 https://github.com/jscl-project/jscl/issues/261 https://github.com/jscl-project/jscl/issues/243 https://github.com/jscl-project/jscl/issues/233 https://github.com/jscl-project/jscl/issues/203 https://github.com/jscl-project/jscl/issues/176 https://github.com/jscl-project/jscl/issues/159 https://github.com/jscl-project/jscl/issues/127 To be closed with this branch, but not closing Issues tickets until this branch can build and pass tests. The upstream issues of course will also need to be parcelled out into more manageable patches. https://github.com/romance-ii/jscl/issues/67 https://github.com/romance-ii/jscl/issues/64 # partial https://github.com/romance-ii/jscl/issues/63 https://github.com/romance-ii/jscl/issues/50 https://github.com/romance-ii/jscl/issues/48 https://github.com/romance-ii/jscl/issues/45 https://github.com/romance-ii/jscl/issues/44 https://github.com/romance-ii/jscl/issues/34 https://github.com/romance-ii/jscl/issues/30 https://github.com/jscl-project/jscl/issues/259 https://github.com/jscl-project/jscl/issues/258 https://github.com/jscl-project/jscl/issues/228 https://github.com/jscl-project/jscl/issues/192 # partial? https://github.com/jscl-project/jscl/issues/164 # partial https://github.com/jscl-project/jscl/issues/17 # partial —~ 𝒻𝒾𝓃𝒾𝓈 ~— fix negative hex When radix is being printed, it should appear before the sign ⇒ (let ((*print-radix* t) (*print-base* 16)) (format t "~a" -42)) #x-2A additional type check Grab the GCL ANSI tests Random puttering Still having issues with not-expanded macros and also with SBCL not liking structure-objects that were defined in the JSCL (storage-vector) way. Keep patch to ANSI-Test here. The current patch is from @jackdaniel and will probably make it into the Git version soon enough, but for the moment, I don't expect there to be “enough” things that we need to patch in the test suite itself to want to maintain a fork. More significantly, for things that are so badly broken that we have to patch the test suite (eg: perhaps a reader macro bug or something), the thing to do is to fix JSCL; in the mean time, this patch file can live as a “pebble in the sandal” so that disabled, broken tests are never forgotten completely. Ignores: renamed targets, re-organize file build docs Stabilization, debugging. Trying to get a clean build from which to close out the remaining critical issues and begin parcelling out patches to upstream. Includes some build engineering work and documentation fixes as well as tracking down build failures. two horribly stupid typos documentation mop up various compile-time errors types, macros, and format options Work on the new Type System infrastructure, and normalizing the code between hosted and self-hosting operations Some more work to get macro-expansion at compile-time working properly for cross-compilation Additional format handlers — `~r`, `~:r`, `~@r`, beginning to work on `~{ ~^ ~}` and `~[ ~]` and friends. sketch in support for other hosts ECL support more SBCL-independence fixups; quasi-quote changes did not work Ignore Emacs project-type files FORMAT ~/funcall/ operation Don't error out on forward-declared functions. This seems to be a bit more complicated than necessary, but it has to do with the awful circularity between the type system and the compiler itself. Perhaps if the low-level bindings are factored out into a quite early module, some of this can be eliminated. FSet-SetF does not neet early Eval-When Share JavaScript's MOST-{POSITIVE,NEGATIVE}-FIXNUM The compiler needs to be on the lookout for bignums, so these figures are duplicated at +MOST-{POSITIVE,NEGATIVE}-FIXNUM+ to be visible in the host compiler as well. confusion over (FUNCTION NAME) This was probably more of a problem with formatting the error message, than with the code, but to avoid the mystifying “#’NAME” messages, renamed the parameter here to FN Forward-declare functions & vars from compiler Types fixups: building hierarchy, Typecases Move all the definitions of built-in-types into a single function to establish the types and classes required by the standard (note: many/most of these, particularly conditions and CLOS things, are not actually defined to have any behavior or even be possible to make instances of, but the type names at least will be known to have the proper subtype relationships) Also trying to clean up some horrible error behaviour from [E]TypeCase, which is not yet solved at this commit Misc compiler fixes: - Fix some SetF definitions - Support typing attributes for Storage-Vectors - Move DECLARATION-INFORMATION to JSCL/CLTL2 - Be more conservative about literals that do not (yet) work in JS - Warn about bad Eval-When situations Add a no-op WITH-COMPILATION-UNIT warn unreachable subtypes in TYPECASE *** empty log message *** make declaration match the actual inferred type later Hosted macroexpansion is the bitch The semi-circular-reference hack Æsthetics of compiling DEFTYPE should EVAL BODY but this nasty hack works for a temp patch dumping function pointers … trashy hack Fix JSCL banner (violet-volts-p) out of order merge failure remove git garbage &environment fall-out make sure environment is set properly fix quasi-quote-removal for SBCL to take (ignored) environment arg defstruct: regression: strings Need symbol, not string, in checking struct accessor objects and such reformat JSCL/COMMON-LISP package (step 1) JSCL/COMMON-LISP package (step 2) JavaScript emulation insanity. The JSCL/CL madness now survives the first compiler pass. need KEYWORD before even COMMON-LISP JSCL/CL package; CONCATENATE Move to JSCL/CL package Also fix CONCATENATE to be a little bit closer to ANSI (handle a few cases that it could not before) — still much to be done. JSCL/CL; VECTOR-PUSH-EXTEND fixes Move to JSCL/CL package A few fixes (untested) for VECTOR-PUSH-EXTEND to get a bit closer to ANSI requirements JSCL/CL and fake generic functions Paving the way for Gray streams implementation using some very shoddy fake generic functions. Actually, this is more like a C++ “virtual dispatch table” (vtbl) here, but it should™ pave the way for proper Gray streams once DefGeneric is in place. Console output class Partial implementation of the web console output as a “class” of stream using the cheezy fake generic functions. Needs to be properly “subclassed” by hacking it into the type system, and the REPL's will also need to be patched to work like this. Work-in-progress. fixups for web-console-output-stream missing arg JSCL/CL and title-case char names Breaks with SBCL's method of printing char names in UPPER-CASE but matches with the preference Mr Steele offers (and I prefer) in CLtL2 — char names like "Latin_Capital_Letter_L" rather than "LATIN_CAPITAL_LETTER_L". They're case-insensitive, regardless. Also switched to the easier-to-follow hex values for Unicode ranges now that we have (for some time now had) #x reader “macro” (actually still a hard-coded hack) support. build more modules in :both define vars and cons functions Just maps car, cdr, rplaca, etc to the host's versions. Not re-inventing that particular wheel if we don't have to. Also trying to recycle a Ruby compiler to perform “var lifting” such that something like: function () { blah(); var x = foo(); blah(x); } … will be transformed into … (lambda () (let (x) (blah) (setq x (foo)) (blah x))) Probably needs some work on real JS scoping rules, but might work well enough for our generated code. fix up JSCL/CL revisions CL Compile-File JSCL/CL renaming and JSCL/JS emulation This branch's problems with weird naming conflicts led to my moving forward with this bit of bootstrapping madness. The changes now in progress here are to create a couple of new packages, which have lives of their own. First Pass: Most of JSCL will first be compiled into JSCL/HOSTED, which uses the code in compat.lisp (JSCL/JS functions) to emulate some of the behaviors of the code generator. In other words, we pass through the compiler to the same JSCL/JS low-level primitives that we would use to generate JavaScript code — but, then, we stop short, and instead populate JSCL/HOSTED. The source code refers only to the JSCL/CL package. At the time this first pass runs, JSCL/CL is JSCL/HOSTED. Second Pass: Next, we invoke the hosted code, and use it to read back in its own source code — only, we don't load the compatibility layer, and we *do* load the code generator. At this point, we are reading in and generating code, still aimed at a JSCL/CL package. Pivot: Once that JavaScript code has been written, we'll “pivot” the package name, making the JSCL/CL package now the COMMON-LISP package. (Possible): Refined self-hosting pass?: In order to clear out any debris from the hosted pass, it might also be wise to run (via Node.js, presumably) a “self-hosted” pure (third) pass, in which the source code is (once again) recompiled by JSCL, but now running natively. If nothing else, the quine-ness of it should be a nice proof-of-concept (and who doesn't love a self-hosting compiler?) Realistically … there may be more Node.js installations in the world than SBCL right now … so the initial bootstrap phase could end up being of less use than the self-hosted version … At this point, much of the work to survive the first pass has been done. The rest remains … misc fiddling doc.css Fix loading FASLs during first-stage bootstrap Load “misc” nearer the end; nothing (much?) depends upon it Fix jscl-repl name to match JSCL/JS emulate MAP-FOR-IN with MAPHASH JSCL/JS fix IN, add NEW Order of operands of JSCL/JS::IN were reversed Clean up symbol-name-printing function slightly Also move some comments into docstrings Clean up compiler errors Moved CL symbol exports table to boot.lisp “You should have received a copy…” Oops! here it is Fix up names for auto-credits UI/cosmetic changes for credits, &c Gather git commit; present credits; comply with the GPL “advertising” requirements for “interactive mode” in the REPL toplevel. Also trap and print errors in EVAL-INTERACTIVE (pending any debugger) Unicode typography Don't be stupid finding src-dir Until we get to ASDF, it's a pain to find the source-dir, but these functions were getting totally confused when Slime ran things in /tmp from hitting <Execute> on individual functions, because *load-pathname* would be set "/tmp/" and we'd do things like trying to get git revisions of your temp-dir (fun…) or read the GPL text from there (who knows, maybe someone left a copy there…) ♯J improvements; use reader macro function in JSCL reader In this patch, moved the CL (compat.lisp) ♯J reader into read.lisp, where it can now be used in both models (SBCL/CL:READ or JSCL/CL:READ). (NOTE: Using actual sharp-sign ♯ here so Git will not treat the octothorpe number-sign as a comment marker.) The following is the revised documentation on the WITH-SHARP-J macro, which is the best description right now. Note the new comma/at-sign forms described below. ———————————————————— The ♯J macro for reading JavaScript forms: ♯J:a:b:c is equivalent to (JSCL/FFI:OGET* *ROOT* \"a\" \"b\" \"c\"), which in turn is equivalent to the JavaScript window[\"a\"][\"b\"][\"c\"] or (when the keys are valid identifiers) also the same as window.a.b.c and a.b.c. Note that the ♯J: sequence is \emph{case-sensitive} and begins with a colon (:) after the ♯J. Each :-delimited substring is the name of a key on the preceding object in the sequence. TODO: An equivalent form where the reference object does not have to be *ROOT* or where identifiers may not have to be strings. There is a discussion in http://github.com/jscl-project/jscl/Issues/ An experimental version is incorporated here. In this notation: Begin with “♯J”. Follow this with : to refer to the root object (normally “window”). Follow it instead with @ to NOT refer to the root object, but instead, name an object (locally in scope) to be referenced. For example, ♯J@foo is identical to FOO. Each subsequent section is introduced by either a colon or comma. A section introduced by : is treated as a string name. A section introduced by a , is instead the name of a locally available (current package) symbol, which is bound to the string value to be used as a key. Examples: ♯J:foo:bar → window[\"foo\"][\"bar\"] ♯J@foo:bar → foo[\"bar\"] ♯J:foo,bar → window[\"foo\"][ bar ] ♯J@foo,bar → foo[ bar ] Note that @ in the first position BOTH suppresses the automatic use of the root object (window) AND ALSO acts like a comma, “unquoting” the following argument. Note that the @ and , options are EXPERIMENTAL and local to the Romance Ⅱbranch. The discussion upstream may substantially change the meaning of these options. Rationale: @ and , are used in Lisp as quasi-quote operators, and are not valid JavaScript identifier components, so they are unlikely to occur in either language in normal use. NOT-TMP: Don't choke on NIL pathnames SRC-DIR: fix sloppy code Handle emulating JS “new” and “this” better Improve MAKE-NEW to invoke constructors Bind JSCL/JS::THIS as a dynamic variable for JS “this” (or should it be literally JSCL/JS::|this|?) Define a p-list-based constructor for |Object|, which is probably the wrong thing to do, but time will tell soon enough. Fixes to ♯J reader code Split up compatibility code for bootstrap The compat.lisp code was getting kinda big. Splitting it up into four separate files just for maintenance purposes. Move some things back from compile-file Collecting some utility functions back in jscl.lisp; perhaps these all belong in some kind of utility file. Some of them can later be factored out in favour of Quicklisp libraries or even just UIOP functions. ignore VAR (we know it's JSCL/JS:VAR) add OSET* to compat-ffi Clean up LOAD-JSCL merge must load compat-sv also unused var removed ignore unused param Fix JSCL/CL package for types.lisp intern type symbols in JSCL/CL JSCL/CL in return-from also one random bug in STRING/= (s2 for n2) CHECK-TYPE: string name for compound types Until we can pretty-print it better, compound types can at least display as their form (eg: (AND INTEGER FIXNUM)) Previously signaled an error trying to coerce the list to a string directly Wishlist XXX — actually render a nice description remove stray symbol Fix bug in ♯J reader String & Object ctors in *ROOT* package names on Return-From's Same code, emulation or not formatting only Re-order compat and try to load each FASL remove Define-CL-Fun finally convert J-Reader to Loop syntax Ran into some quirkiness with the more obvious for start fixnum = 1 then (1+ end) and end = (position-if … ) … thus the slightly less beautiful SetQ This is, to my eyes, easier to follow than a complex (Do … ) form … More for completeness than usefulness fixup, read-char in #* reader “macro” EMPTYP is not in :CL comments Clarify meaning of error message also put nice warning icons list the names in the CError add format # option I honestly had no idea this existed, and now feel foolish at having done apparently unnecessary things more than once to have the same effect. “In place of a prefix parameter to a directive, … You may also use the character # in place of a parameter; it represents the number of arguments remaining to be processed.” use top-level eval in hosted mode? Duplicate T/NIL? Ensure *ROOT* is bound search for types in global environment also reformat reindent Move symbols to JSCL/CL package Fix some compiler warnings in SBCL Note, DO-SEQUENCE signals warnings once the arg has already been inferred to be a list or array (by the etypecase) because it generates code for both cases, so one branch is known to always be never taken — the macro could be improved, but just using DOLIST and DOTIMES directly was easier. Fix up JSCL/CL symbols emulate lisp-to-js (no-op) emulate OSET as (SETF OGET) Move HASH-TABLE to JSCL/CL; use storage-vectors The storage-vector implementation here is really just to take advantage of the KIND field, and still stuffs everything into a JS Object; ie, this is a (VECTOR 1 JS:Object) with a KIND property attached. To make the KIND more meaningful, it's stored always as a symbol (not a function directly); ie, (make-hash-table :test #'eql) is converted backwards to yield a kind of (HASH-TABLE EQL) — the symbol (QUOTE EQL) not the function-object (FUNCTION EQL). EQUALP is still a FIXME/TODO, for now. Export some MOP symbols fixup typo fix up symbol packages fix typecase bug in duplicate detection TYPECASE does not imply (OR…) I don't know why I had a bout of Stupid when I wrote this, but TYPECASE isn't like CASE; if you want (OR TYPE1 TYPE2) you have to spell it out. Do-All-External-Symbols is not in CL Move to JSCL/CL; clean DESCRIBE Take advantage of nascient type system and honour the STREAM parameter; also use ~vT and *Print-Right-Margin* for formatting. Output of DESCRIBE changed a bit due to these changes. NB: TODO: The REPL should set *Print-Right-Margin* the way Slime does, probably hooking into window resize events or something. Clean up some utility functions Initial DEFCLASS sketch reformat remove split-up file reformat reformat begin CLOSsifying conditions fixups for class integration methods as classes multiple initarg aliases reformat format ~:[ and ~@[ partial finding COPYING file CONDITION-P, fixes namespace FSet run-command incovation host name, not domain name JSCL/JS::FSET emulation warnings re-order in dependency order log of rebuild errors remove log stash fixup of “real” ERROR Leave the definition in boot.lisp for early in the bootstrap process, only ERROR coöperate with conditions.lisp Don't revert to the poorer definition, in case the better one in conditions.lisp may have been handled already. fix call to REDUCE EQL-Specializer-p is in MOP package ignore log files correct comments also unused field format remove unwanted macro redefinition legacy of before moving to JSCL/CL package unnecessary package prefices re-organize ordering move function define all standard classes Note that this introduces a FIXME on the class of these types. don't stop if it didn't fail fixes to reporting warnings/failures fix docstring get-internal-real-time log builds always report warnings/failures fixups to type-of Breaking out bootstrap package also comments ctags eliminate compiler warnings ambiguity in `Do-Source' over-optimized loop in `Git-Credits' Export bootstrapping from JSCL/Bootstrap package fix comment ensure the first defpackage gets evaluated at load-time missing In-Package moved fn package fix up package renames package renames package renamed fixups renaming packages for second pass: fix label file compile-file: returning three values ignore condition object spurious space load FASLs only on host pass clarify messages with in which pass errors occur not-yet-working cross-compilation (second pass) symbols not yet defined Use "intern" so code loads jscl:test-files not yet defined Use (primitive) IF before macro UNLESS with-compilation-environment fixups to README Don't allow JS reserved words as identifiers The rather long list of reserved words in JavaScript should never be used for identifiers. That blacklist, from https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Lexical_grammar is now incorporated into Valid-JS-Identifier. valid-js-identifier-p (add -p) Fix up bootstrap package name in Makefile Export Load-JSCL fix whitespace errors Try to recover from aborted loads APPLY constructor to args (not FUNCALL) ensure creation of KEYWORD package (needed by Reader) fix emulation of JS Symbol ctor SBCL, at least, doesn't support :if-exists :new-version This is probably true for all Unix-based Lisps, since the tradition is to use ~n~ or a VCS on Unix/POSIX/Linux systems, rather than actual file versions; but WebDAV *does* versioning, so JSCL should support it once the DAV-based filesystem (at least) is in place. fixup unwind-protect forms around second-stage cross-compilation fix indentation better recovery from failures in Load-JSCL give up on deleting the current package change JSCL/JS::Var macro remove unused var rename “this” as “*This*” package designator is a string designator package creation in bootstrap %make-package work package of JSCL/JS symbols reformat JSCL/Impl package No second-pass package renaming clean up package refs verbose messaging unnecessary distinction fix up boot.lisp package creation DESCRIBE: ignore errors for non-type/class; mention special forms Trying (failing) to ensure Special Forms are processed The CONVERT-* family of functions are not catching special forms like LET and ∴ the compilation crashes by incorrectly believing them to be function calls, and trying to evaluate the variable binding forms as function calls. EG: (let ((a b)) a) → (funcall ((a b)) a) → error, (a b) is not a fn designator Ensure CL symbols are exported in both hosted JSCL and host CL hash-table EQ map symbols twice fix assertions in ~:[ ~; ~] fix name interning for define-compilation teletype-p for ECL Thanks @jackdaniel ECL stream-clear-output special forms the special-forms/macros bug crashing special-forms storage-vector literals merge special-operator-p fix assert format string qualified name on type in FTYPE initargs (plural) reading structures return the storage-vector fix Literal-SV missing defvar (in build order) fix messages change order so pathnames are caught sooner reorder wip
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
No description provided.
The text was updated successfully, but these errors were encountered: