--native-lib

[ncannasse]

While we're changing the commandline implementation, I think we should review our definition of "native library".

ATM we have:

• haxelib path can return -L for native neko libs. There's no other way to inject them
• Flash have -swf-lib, C# has -net-lib , Java has -java-lib

I propose regrouping all of the there under a single --native-lib (with aliases to older -swf-lib etc.), and have a single com.native_libs.

[RealyUniqueName]

Let's make --native-lib work for other targets as well. That would be consistent.
E.g. it can behave like Compiler.includeFile() for lua and js.
Python and php could import native files.
Maybe dll/so for C++ if it's possible.

[back2dos]

What does haxe.macro.Compiler.addNativeLib do right now though? ^^

[ncannasse]

https://github.com/HaxeFoundation/haxe/blob/development/src/macro/macroApi.ml#L1860

…

On Sat, Sep 22, 2018 at 7:33 AM Juraj Kirchheim ***@***.***> wrote: What does haxe.macro.Compiler.addNativeLib do right now though? ^^ — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7450 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA-bwLiZpx21h7zjMFiYKUCYrn2Pemrvks5udcuWgaJpZM4WzsyP> .

[Simn]

I think we have to do something here for 4.0. I'm not just talking about --native-lib as a unified flag, but also about the native library vs. compilation server situation.

Related issues: #7417 and #7651

Our native library handling was never updated to support the compilation server properly. On every request, which includes display requests, the functionality kicks in and does something.

The way it currently works is this:

1. Open and parse the native library file
2. Set up a lookup to resolve dot-paths to parsed modules (in com.load_extern_type)

It's important to note that step 2 is lazy: The native representation is not converted to a Haxe module unless explicitly requested via dot-path.

The problem is that we perform step 1 on every request. Even worse, C# actually goes on a hunt in the file system to find its net-std file, which is part of the problem in #7651.

---

I propose the following approach when the compilation server is running:

• For each context, keep track of which native library files have been processed. Also track the timestamp so we can detect file changes.
• When processing --native-lib, check that cache. If the file has been processed, do nothing.
• If it has not, perform the native -> Haxe conversion eagerly for all types. This is a bit expensive, but we only have to do it once per context. Because we're only storing the parsed representation, this should not lead to a huge memory usage increase.

This would greatly simplify the handling on the compilation server because native libraries become a one-time read-and-forget thing. It will automatically fix related display issues like #7417 because we deal with the syntax representation as if we had normal Haxe modules.

We can still keep load_extern_type around for the offline mode because there lazy loading certainly makes sense.

Any thoughts?

[Aurel300]

What if the eager loading finds some types in a library that cause an error? With lazy loading you can simply avoid importing/using these types.

[Simn]

Technically, that shouldn't happen in the first place. But putting a catch-all around this might be a good idea regardless.

[Simn]

My approach doesn't work because the parser-cache is file-based. This would require a separate caching mechanism, which makes matters much more complicated.

[ncannasse]

We don't really need eager evaluation, just keeping the modules in memory and linking them to the native library as a dependency should make sure they are kept in cache and only discarded if native library changes. This would still reparse the library in case a _new_ module gets loaded from it, but we can also have a cache for the step 1

…

On Mon, Aug 12, 2019 at 4:02 PM Simon Krajewski ***@***.***> wrote: My approach doesn't work because the parser-cache is file-based. This would require a separate caching mechanism, which makes matters much more complicated. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#7450?email_source=notifications&email_token=AAHZXQCXVFNKT2OWIXBIETLQEFUN5A5CNFSM4FWOZSH2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4CT6ZA#issuecomment-520437604>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAHZXQDSQUADF35U3P5QXRLQEFUN5ANCNFSM4FWOZSHQ> .

[Simn]

I'm not quite sure what you're saying we should cache. There are 4 stages when it comes to processing a native library:

1. Read the data from the filesystem
2. Convert it to the native representation
3. Convert that to Haxe AST
4. Type the Haxe AST

I'm proposing to cache 3. because that's what the display handling needs and I doubt there's much of a size difference between 2. and 3.

[Simn]

Regarding the actual new CLI argument, there's the need to configure some things per-library (e.g. a flag to ignore unknown types in #3397, or whether to include a native library in the output).

How about we allow JSON input for this new flag? It would be --native-lib '{ "file": "lib/whatever.jar", "ignoreUnknownTypes": true, "includeInOutput": false }'. We can still accept --native-lib lib/whatever.jar for simplicity.

[RealyUniqueName]

Having to type json in CLI args makes me sad.
Can't we have --native-lib-file, --native-lib-ignore-unknown-type etc? Or -D flags?

[Simn]

How would that work in practice given that the setting is per-library?

[back2dos]

What's the relationship between the compiler arg and Compiler.addNativeLib? Could we rely on adding optional args to the latter to handle the more complex use cases?

If that's not an option, I would suggest that the --native-lib may in fact be a JSON file, that contains the config (including relative path to the actual .jar/.dll (should probably default to a sensible convention)). So instead of --native-lib '{ "file": "lib/whatever.jar", "ignoreUnknownTypes": true, "includeInOutput": false }' one would write --native-lib lib/whatever.json with that file containing { "ignoreUnknownTypes": true, "includeInOutput": false }.

---

On a side note: I wonder why we're using JSON for everything lately. For some of this config stuff, I think haxe expressions might be a better fit. Not only are they nicer to write (less quotes, allow comments, trailing commas, could even contain enums where it makes sense), we also have a position tracking parser for them. And I suspect the typer could be leveraged to guarantee validity (would probably need a separate context to not cause mayhem - perhaps there's a way to eagerly populate it with types and then disable further type loading it it?) and give things like unknown field ingoreUnknownTypes, (Suggestion: ignoreUnknownTypes) with exact positions.

[Simn]

I like the idea to allow --native-lib lib/whatever.json. That keeps the CLI clean and still allows to convey all the required information.