Skip to content

Commit

Permalink
Hxb (#11504)
Browse files Browse the repository at this point in the history
* use identity for field params

* support partial generic expansion for fields

* take off @:generic

* try something for enum constructor type params

* gencommon says no

* communicate field type parameters by index again

* surely anons are just a list of anon field refs

* fix generic type param names

* let's get dangerous

* handle CBOs in HxbRestore too

* well

* chdir

* handle warnings as CBOs

* don't try to hxb import.hx because there's nothing in there

* don't roundtrip on Cross and Eval

way too many false positives in tests from printing

* add CFLR, don't resolve fields like a crazy person

* fix but it's still broken

* stop the roundtrip nonsense for now, enough other problems

* decode binary modules in check_display_file

* remove flush_fields, should not be needed with CFLR

* maybe fix generic type parameter naming

* add ENFR too

* pull stricter tanon changes and remove some debug

* unused open

* fix overloads a bit more

but it's still not 100% working

* fix generic type param naming a bit more

* check for cf_overloads instead of CfOverloads

The flag isn't set for ghetto overloads, so this should catch all relevant cases.

* fix overloads even more

* add enum when adding enum field ref

* fix overloads for real

* small optimization for the VERY common depth = 0 case

* properly initialize typedef monos

* map anons before identifying them

* write tpp paths instead of names

* remove some debuggery from the writer

* embrace ttp

* don't follow away Null<Void> in function returns

* encode basic types directly

I'm not sure if I want to keep this optimization, but the changes to com.basic seem like a good idea regardless. At the moment it might be possible to get a hold of the ominous `m` monomorph in Common.create, and binding that one to anything would lead to interesting results.

* add ANFR for anon fields

* Happy new year!

* [typer] fix functional interface type parameter leak

see #11390

* ANFR earlier

* only write cf_expr_unoptimized if it differs from cf_expr

* write texpr positions are bit smarter

* add field contexts, hashcons expression type instances

* [display] populate server/modules from hxb since there's only that atm

* remove cf_expr_unoptimized optimization again because the hashconsing has already broken it

We need a different solution for this silly field because it takes up a lot of space and time for very little gain. Maybe something like a binary diff to cf_expr would make sense here.

* forward declare locals in texpr

way less Hashtbling

* be less awkward around ttps

They're your friend!

* purge some debug from the reader

We never want to write to stderr because that hangs the display tests.

* don't generate TLazy and remove random bool

Also add comment about empty anons

* make api an argument of the read function

This is a prerequisite for making the reader reentrant, which should now be possible.

* demo how continuations could work

* reorder tables to something that makes sense to me

* separate chunk classes more

* add cl_type to avoid hundreds of ClassStatics anons

* don't set anon fields too early

* properly deal with ClassStatics in unification and field typing

also convert unify_anons to something human-readable

* bring back missing checks

* add small ring cache per type kind

* add more elaborate stats behind -D hxb.stats

* move expr stuff out of the way because I keep scrolling past it

* change mono byte to 1, avoid extra 1 byte for immediate type instance values

* encode simple type instances immediately

* potentially avoid some GC write barriers by splitting up pos

* use less ignorant fast_eq

* warnings

* basic reader stats

* minor cleanup

* fix class field scoping

* infer nested status from stack

Did you know there can be anon fields inside enum fields? Crazy...

* field type parameters 2.0

* put all texpr type instances in an array

* change -bcp to --hxb-lib

* create directory before creating file...

* format stats output nicer because that's very important

* rename hxml so I find it easier

* encode Void directly because it actually appears a lot

* reorganize type instance bytes, inline common list lengths

* warnings

* write blocks less awkwardly

also don't read arrays only to turn them into lists

* don't write i32 if it can be helped

* pass current chunk to pos writer

* add texpr stats

* optimize TCall

* optimize static calls

* refactor pos writer slightly

* optimize this.field

* fix generator printing

* enable disabled tests

* remove overly fancy position optimization

* write TInt as leb too

* Revert "write TInt as leb too"

This reverts commit ce6c8de.

* let's see where we're at

* check context before checking cache for display files

see #11480

* stop leaving debug prints in your commits ffs

* adjust test

Calling exclude before saving means that the class is extern on the next iteration, and thus not generated.

* dodge arcane sourcemap test problem

* dodge more

* change chunk to module in the writer

* use SimnBuffer

* remove roundtrip

* avoid some object overhead in the reader

* use correct v_id when reading from .hxb

* [hxb] use cache hxb for --hxb too

* don't generate aux output (including hxb) during diagnostics

* warnings

* only cache context when using compilation server

* write chunk name before length, and remove crc

The 4 bytes of the name don't contribute to the length, so this makes more sense.

* minor refactor to make interface a bit saner

* skip hxb generation on display too

* warnings

* move module_cache to HxbData

That might not be the ideal place for it either but it's better than tType.ml

* rework reader interface a bit as well

* read metadata in forward data

see go2hx/go2hx#174

* read anon field meta too

* persist ordered list of chunks instead of one big bytes thing

* remove feature nonsense and revert meta change

* warnings

* fix server/module, get all data from hxb cache directly

* align chunk order

* [hxb] don't overwrite taint reason with check_display_file

* [server] we're sending cacheState, not 'dirty'

* [tests] update test

* [hxb] generate proper warnings for unbound type parameters

* Simplify unbound ttp positions; not like they're often useful anyway...

* ocaml syntax is hard

* anon fields can have @:overload too...

* use singular empty anon

* slightly improve field vs. overloads handling

* remove some debug because hxb is perfect

* [TYPF] skip module name, only write pack when different from current module (private types)

* no need to complicate things

* [tests] add test for #11480

* add CFEX, but don't generate anything into it yet

* move local type params to expressions

* don't write type parameter length twice

* write ttp host

* write nested status so the reader doesn't have to track this

* more CFEX work

* rename chunks

* move all field references in front of definitions

There's currently no data dependency which requires this, but it's more future-proof in case there's ever something along the lines of dependent types.

* rename chunk reader functions too

* sanity catch Error when resolving types

* more sanity

* forward declare module type parameters in MTF

We need to know their identity early so the instance builder can work. Constraints and defaults are set later.

* fix overload handling again

* remove manual cl_build call, add enable_field_access to api

see #11493

* add marker chunks, read hxb modules delayed

see #11493

* work around java.Init issue so we can run JVM roundtrip on CI

see #11493

* fail nicer if we can't write the archive

* write cl_flags in MTF

We want to know early if a class is an interface

* remove enable_field_access, rely on typing passes instead

see #11493

* only load a native lib once we actually need it

* generate extern modules too

* activate EXD, handle field type parameters awfully for now

* don't write empty chunks

* reduce diff against development

* bucket class fields by their name to avoid long linear lookups

* remove rings

Not worth it anymore because the type instance simple/not_simple distinction is fast enough.

* comment out stats writes because this might be an observable overhead

* time writer per-module so we can find pathological modules

* lose some more byte-level OOP overhead

* remove some debug noise

* merge IOChunk and Chunk, remove string_pool

* no comment

* simplify type instance writing again

Reuse a single chunk that we reset when writing a texpr type instance.

* add custom StringPool implementation

* optimize locals a bit more

* maybe deal with duplicate var declaration

* revert local changes

* try local optimization again

* invert texpr order: kind, type, pos

This will allow us to omit some type instances in case they can be inferred from the kind.

* introduce implicit texpr type instances

* walk back a little

* don't double match type params

* avoid some EXD reading

* small cleanup

* avoid some list reading for things that aren't lists

* reverse field order and create both list and PMap at the same time

* Generate dump even with --no-output

* avoid write_list for anon fields too

Folding a PMap to a list only to then get its length (which is a linear operation) and iterate over it isn't the power move that I thought it was.

* write pos pairs where applicable

* delay IO.input creation, add per-chunk timer

* avoid IO in reader too

* don't look if you want to sleep at night

* less nightmares

* remove abstract reader

* do less in display requests

CI exploding in 3... 2... 1...

* start working on delayed expression reading

see #11498

* Add hxb.stats to ignored defines for signature

* put the dodge in place, activate

see #11498

* adjust to cl_init changes

* used hashed identity pool for anon fields too

* less classes

* less classes

* no classes

* avoid some pointless DynArray to List operations

* more DynArray

* add MDR to deal with import to @:keep

* write CLR and friends after CFR

CFR might add more CLR. Also add sanity checks to make sure we don't modify pools after exporting them.

* actually install cf_type

* change field type parameter storing to something more awkward and efficient

* avoid some unnecessary byte blitting

* fix HxbId insertion on fields without expression

* Warnings

* [server-hxb] only read expressions eagerly when full typing

* [hxb] write type for TMeta

* [hxb] whitespace nazi

* tanon identification / tunification: stricterer EqStricter

* Why was this debug even pushed..

* hacktoberfest

* [display] server/type: only restore hxb headers

* check if 3725 is still a problem

* remove TODO comment

* Remove TODOs

* clean up unit hxmls

* don't null-terminate bytes

* [tests] add display test for static field completion

Also run again a couple tests after caching type

* die when writing out empty module or type name

The reader would die anyway

---------

Co-authored-by: Rudy Ges <k@klabz.org>
  • Loading branch information
Simn and kLabz committed Jan 25, 2024
1 parent be88d42 commit fe395ef
Show file tree
Hide file tree
Showing 31 changed files with 5,083 additions and 77 deletions.
6 changes: 6 additions & 0 deletions src-json/meta.json
Original file line number Diff line number Diff line change
Expand Up @@ -518,6 +518,12 @@
"platforms": ["hl"],
"targets": ["TClass", "TClassField"]
},
{
"name": "HxbId",
"metadata": ":hxb.id",
"doc": "Internally used by hxb",
"internal": true
},
{
"name": "HxCompletion",
"metadata": ":hx.completion",
Expand Down
12 changes: 10 additions & 2 deletions src-json/warning.json
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,14 @@
"name": "WConstructorInliningCancelled",
"doc": "Constructor call could not be inlined because a field is uninitialized",
"parent": "WTyper"
},
{
"name": "WHxb",
"doc": "Hxb (either --hxb output or haxe compiler cache) related warnings"
},
{
"name": "WUnboundTypeParameter",
"doc": "Hxb (either --hxb output or haxe compiler cache) failed to link a type parameter to an actual type",
"parent": "WHxb"
}

]
]
2 changes: 1 addition & 1 deletion src/codegen/genxml.ml
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ let rec follow_param t =
t

let gen_meta meta =
let meta = List.filter (fun (m,_,_) -> match m with Meta.Used | Meta.RealPath | Meta.Pure -> false | _ -> true) meta in
let meta = List.filter (fun (m,_,_) -> match m with Meta.Used | Meta.RealPath | Meta.Pure | Meta.HxbId -> false | _ -> true) meta in
match meta with
| [] -> []
| _ ->
Expand Down
9 changes: 9 additions & 0 deletions src/compiler/args.ml
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ let parse_args com =
let actx = {
classes = [([],"Std")];
xml_out = None;
hxb_out = None;
json_out = None;
cmds = [];
config_macros = [];
Expand All @@ -58,6 +59,7 @@ let parse_args com =
interp = false;
jvm_flag = false;
swf_version = false;
hxb_libs = [];
native_libs = [];
raise_usage = (fun () -> ());
display_arg = None;
Expand Down Expand Up @@ -131,6 +133,10 @@ let parse_args com =
("Compilation",[],["-libcp"],Arg.String (fun path ->
com.class_paths#add (new ClassPath.directory_class_path (Path.add_trailing_slash path) Lib);
),"<path>","add a directory to find source files");
("Compilation",["--hxb-lib"],["-hxb-lib"],Arg.String (fun file ->
let lib = create_native_lib file false HxbLib in
actx.hxb_libs <- lib :: actx.hxb_libs
),"<path>","add a hxb library");
("Compilation",["-m";"--main"],["-main"],Arg.String (fun cl ->
if com.main_class <> None then raise (Arg.Bad "Multiple --main classes specified");
let cpath = Path.parse_type_path cl in
Expand Down Expand Up @@ -272,6 +278,9 @@ let parse_args com =
("Services",["--json"],[],Arg.String (fun file ->
actx.json_out <- Some file
),"<file>","generate JSON types description");
("Services",["--hxb"],[], Arg.String (fun dir ->
actx.hxb_out <- Some dir;
),"<directory>", "generate haxe binary representation in target directory");
("Optimization",["--no-output"],[], Arg.Unit (fun() -> actx.no_output <- true),"","compiles but does not generate any file");
("Debug",["--times"],[], Arg.Unit (fun() -> Timer.measure_times := true),"","measure compilation times");
("Optimization",["--no-inline"],[],Arg.Unit (fun () ->
Expand Down
58 changes: 45 additions & 13 deletions src/compiler/compilationCache.ml
Original file line number Diff line number Diff line change
Expand Up @@ -23,9 +23,18 @@ type cached_native_lib = {
c_nl_files : (path,Ast.package) Hashtbl.t;
}

class context_cache (index : int) = object(self)
let get_module_name_of_cfile file cfile = match cfile.c_module_name with
| None ->
let name = Path.module_name_of_file file in
cfile.c_module_name <- Some name;
name
| Some name ->
name

class context_cache (index : int) (sign : Digest.t) = object(self)
val files : (Path.UniqueKey.t,cached_file) Hashtbl.t = Hashtbl.create 0
val modules : (path,module_def) Hashtbl.t = Hashtbl.create 0
val binary_cache : (path,HxbData.module_cache) Hashtbl.t = Hashtbl.create 0
val removed_files = Hashtbl.create 0
val mutable json = JNull
val mutable initialized = false
Expand Down Expand Up @@ -57,17 +66,41 @@ class context_cache (index : int) = object(self)
method find_module_opt path =
Hashtbl.find_opt modules path

method cache_module path value =
Hashtbl.replace modules path value
method find_module_extra path =
try (Hashtbl.find modules path).m_extra with Not_found -> (Hashtbl.find binary_cache path).mc_extra

method cache_module warn anon_identification hxb_writer_stats path m =
match m.m_extra.m_kind with
| MImport ->
Hashtbl.add modules m.m_path m
| _ ->
let writer = HxbWriter.create warn anon_identification hxb_writer_stats in
HxbWriter.write_module writer m;
let chunks = HxbWriter.get_chunks writer in
Hashtbl.replace binary_cache path {
mc_path = path;
mc_id = m.m_id;
mc_chunks = chunks;
mc_extra = { m.m_extra with m_cache_state = MSGood }
}

method clear_cache =
Hashtbl.clear modules

(* initialization *)

method is_initialized = initialized
method set_initialized value = initialized <- value

method get_sign = sign
method get_index = index
method get_files = files
method get_modules = modules

method get_hxb = binary_cache
method get_hxb_module path = Hashtbl.find binary_cache path

(* TODO handle hxb cache there too *)
method get_removed_files = removed_files

method get_json = json
Expand Down Expand Up @@ -115,7 +148,7 @@ class cache = object(self)
try
Hashtbl.find contexts sign
with Not_found ->
let cache = new context_cache (Hashtbl.length contexts) in
let cache = new context_cache (Hashtbl.length contexts) sign in
context_list <- cache :: context_list;
Hashtbl.add contexts sign cache;
cache
Expand Down Expand Up @@ -174,7 +207,14 @@ class cache = object(self)
Hashtbl.iter (fun _ cc ->
Hashtbl.iter (fun _ m ->
if Path.UniqueKey.lazy_key m.m_extra.m_file = file_key then m.m_extra.m_cache_state <- MSBad (Tainted reason)
) cc#get_modules
) cc#get_modules;
let open HxbData in
Hashtbl.iter (fun _ mc ->
if Path.UniqueKey.lazy_key mc.mc_extra.m_file = file_key then
mc.mc_extra.m_cache_state <- match reason, mc.mc_extra.m_cache_state with
| CheckDisplayFile, (MSBad _ as state) -> state
| _ -> MSBad (Tainted reason)
) cc#get_hxb
) contexts

(* haxelibs *)
Expand Down Expand Up @@ -267,11 +307,3 @@ type context_options =
| NormalContext
| MacroContext
| NormalAndMacroContext

let get_module_name_of_cfile file cfile = match cfile.c_module_name with
| None ->
let name = Path.module_name_of_file file in
cfile.c_module_name <- Some name;
name
| Some name ->
name
4 changes: 4 additions & 0 deletions src/compiler/compilationContext.ml
Original file line number Diff line number Diff line change
Expand Up @@ -11,6 +11,7 @@ type native_lib_kind =
| NetLib
| JavaLib
| SwfLib
| HxbLib

type native_lib_arg = {
lib_file : string;
Expand All @@ -21,6 +22,7 @@ type native_lib_arg = {
type arg_context = {
mutable classes : Globals.path list;
mutable xml_out : string option;
mutable hxb_out : string option;
mutable json_out : string option;
mutable cmds : string list;
mutable config_macros : string list;
Expand All @@ -31,6 +33,7 @@ type arg_context = {
mutable interp : bool;
mutable jvm_flag : bool;
mutable swf_version : bool;
mutable hxb_libs : native_lib_arg list;
mutable native_libs : native_lib_arg list;
mutable raise_usage : unit -> unit;
mutable display_arg : string option;
Expand All @@ -56,6 +59,7 @@ and compilation_context = {
type compilation_callbacks = {
before_anything : compilation_context -> unit;
after_target_init : compilation_context -> unit;
after_save : compilation_context -> unit;
after_compilation : compilation_context -> unit;
}

Expand Down
9 changes: 8 additions & 1 deletion src/compiler/compiler.ml
Original file line number Diff line number Diff line change
Expand Up @@ -295,6 +295,7 @@ let do_type ctx mctx actx display_file_dot_path =
CommonCache.maybe_add_context_sign cs com "before_init_macros";
enter_stage com CInitMacrosStart;
ServerMessage.compiler_stage com;
Setup.init_native_libs com actx.hxb_libs;
let mctx = List.fold_left (fun mctx path ->
Some (MacroContext.call_init_macro ctx.com mctx path)
) mctx (List.rev actx.config_macros) in
Expand Down Expand Up @@ -378,14 +379,19 @@ let compile ctx actx callbacks =
let (tctx,display_file_dot_path) = do_type ctx mctx actx display_file_dot_path in
DisplayProcessing.handle_display_after_typing ctx tctx display_file_dot_path;
finalize_typing ctx tctx;
let is_compilation = is_compilation com in
com.callbacks#add_after_save (fun () ->
callbacks.after_save ctx;
if is_compilation then Generate.check_hxb_output ctx actx;
);
if is_diagnostics com then
filter ctx tctx (fun () -> DisplayProcessing.handle_display_after_finalization ctx tctx display_file_dot_path)
else begin
DisplayProcessing.handle_display_after_finalization ctx tctx display_file_dot_path;
filter ctx tctx (fun () -> ());
end;
if ctx.has_error then raise Abort;
if is_compilation com then Generate.check_auxiliary_output com actx;
if is_compilation then Generate.check_auxiliary_output com actx;
enter_stage com CGenerationStart;
ServerMessage.compiler_stage com;
Generate.maybe_generate_dump ctx tctx;
Expand Down Expand Up @@ -446,6 +452,7 @@ with

let finalize ctx =
ctx.comm.flush ctx;
List.iter (fun lib -> lib#close) ctx.com.hxb_libs;
(* In server mode any open libs are closed by the lib_build_task. In offline mode
we should do it here to be safe. *)
if not ctx.comm.is_server then begin
Expand Down
63 changes: 63 additions & 0 deletions src/compiler/generate.ml
Original file line number Diff line number Diff line change
@@ -1,5 +1,7 @@
open Globals
open CompilationContext
open TType
open Tanon_identification

let check_auxiliary_output com actx =
begin match actx.xml_out with
Expand All @@ -19,6 +21,67 @@ let check_auxiliary_output com actx =
Genjson.generate com.types file
end

let export_hxb com cc platform zip m =
let open HxbData in
match m.m_extra.m_kind with
| MCode | MMacro | MFake | MExtern -> begin
(* Printf.eprintf "Export module %s\n" (s_type_path m.m_path); *)
let l = platform :: (fst m.m_path @ [snd m.m_path]) in
let path = (String.concat "/" l) ^ ".hxb" in

try
let hxb_cache = cc#get_hxb_module m.m_path in
let out = IO.output_string () in
write_header out;
List.iter (fun (kind,data) ->
write_chunk_prefix kind (Bytes.length data) out;
IO.nwrite out data
) hxb_cache.mc_chunks;
let data = IO.close_out out in
zip#add_entry data path;
with Not_found ->
let anon_identification = new tanon_identification in
let warn w s p = com.Common.warning w com.warning_options s p in
let writer = HxbWriter.create warn anon_identification com.hxb_writer_stats in
HxbWriter.write_module writer m;
let out = IO.output_string () in
HxbWriter.export writer out;
zip#add_entry (IO.close_out out) path;
end
| _ ->
()

let check_hxb_output ctx actx =
let com = ctx.com in
let try_write path =
let t = Timer.timer ["generate";"hxb"] in
Path.mkdir_from_path path;
let zip = new Zip_output.zip_output path 6 in
let export com =
let cc = CommonCache.get_cache com in
let target = Common.platform_name_macro com in
List.iter (fun m ->
let t = Timer.timer ["generate";"hxb";s_type_path m.m_path] in
Std.finally t (export_hxb com cc target zip) m
) com.modules;
in
Std.finally (fun () ->
zip#close;
t()
) (fun () ->
export com;
Option.may export (com.get_macros());
) ()
in
begin match actx.hxb_out with
| None ->
()
| Some path ->
try
try_write path
with Sys_error s ->
error ctx (Printf.sprintf "Could not write to %s: %s" path s) null_pos
end

let parse_swf_header ctx h = match ExtString.String.nsplit h ":" with
| [width; height; fps] ->
Expand Down

0 comments on commit fe395ef

Please sign in to comment.