Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Haxe-in-Haxe notes #6843

Open
ncannasse opened this Issue Feb 10, 2018 · 23 comments

Comments

Projects
None yet
10 participants
@ncannasse
Copy link
Member

ncannasse commented Feb 10, 2018

The first goal is to have the compiler written in Haxe and generating the corresponding OCaml code, allowing for a more familiar syntax for potential contributors and compiler team. At longer term, this will allow the compiler to run on other platforms although this is not something that will be possible or that we should focus at first.

The following needs to be resolved to have a working Haxe-in-Haxe implementation.

Note to Haxe users : this is an exploration of the possibility, we are not sure yet we will have something working in the end.

OCaml Haxe target (Haxe-to-OCaml) (@ncannasse) :

  • OCaml generator for Haxe compiler: we shouldn't try to support the whole Haxe specification at first, in particular everything involving reflection / dynamic is not used by the compiler and thus not required.
  • Class/structures representation: we should go with something like C, with a data structure representing the classes fields and methods taking the class as first argument.
  • Ability to access externs, such as ExtLib PMap module, etc. This should work pretty will with abstracts or extern classes given the class representation choice
  • List support: not an easy one, we need to have some way to easily construct and pattern match ml immutable lists
  • More list support: :: as proper operator with type inference, correctly infer :: in pattern matching

Automated Haxe compiler OCaml-to-Haxe ML2HX (@nadako) :

  • Extracting the OCaml typed AST
  • Generating the corresponding Haxe code
  • Lexing and parsing: I think they should be left as "externs" for now, and we can keep the original OCaml source
  • Making sure the corresponding Haxe code compiles (gets typed) with the ML target

Merge of both ML target and ML2HX :

  • Making sure the Haxe compiler code successful generates to ML
  • Making sure the ML generated code compiles and run
  • Performance/Styling : replace consecutive var assignment with lets in ML target, replace consecutive lets with same type by single var decl in ML2HX
  • Deal with Lexer and Parser at this point?
  • Refactoring: after porting, several steps of refactoring will be possible by reorganizing the modules and packages
@ncannasse

This comment has been minimized.

Copy link
Member Author

ncannasse commented Feb 10, 2018

I have added :: operator and ability to construct and pattern match immutable lists, this can be followed on this branch https://github.com/HaxeFoundation/haxe/tree/genml

@nanjizal

This comment has been minimized.

Copy link

nanjizal commented Feb 10, 2018

Since Haxe currently support C# would F# provide a simpler path.
http://web.archive.org/web/20080410181630/http://research.microsoft.com/fsharp/manual/ml-compat.aspx
https://stackoverflow.com/questions/179492/f-changes-to-ocaml
The task could then be split between modifying the C# target to support F#, and a task to change the compiler to F# code while keeping information on changes made so that Ocaml can be supported later.
Haxe c# dll could be consumed which would allow you to plugin hard functionality fairly early on and optimise and rewrite as needed. Anyway just an idea I have no idea if it fits the needs but if someone was able to get a minimal F# version of Haxe compiler working quickly it would prove the concept.

@nanjizal

This comment has been minimized.

Copy link

nanjizal commented Feb 10, 2018

I have compiled F# mono graphics examples on mac so it's not just windows although doubt it would provide the same speed as Ocaml initially?

@nanjizal

This comment has been minimized.

Copy link

nanjizal commented Feb 10, 2018

I also wonder about using hxcpp but I suppose libraries like
https://github.com/GJDuck/libf
Don't produce the richness that Ocaml has.

@ncannasse

This comment has been minimized.

Copy link
Member Author

ncannasse commented Feb 10, 2018

@nanjizal I don't think F# produces as fast code as OCaml do, as it's built on top of DotNet

@ncannasse

This comment has been minimized.

Copy link
Member Author

ncannasse commented Feb 10, 2018

@nanjizal anyway I have little interest in targeting F#, and this requires extra profiling and extensive changes as the article explains

@kevinresol

This comment has been minimized.

Copy link
Contributor

kevinresol commented Feb 10, 2018

@nanjizal let's not diverge the focus here. Porting from the compiler from OCaml to Haxe is a big task already. Let keep this post focused on this single aspect.

Using other target language is beyond the scope of the post as said in OP already:

... to run on other platforms ... is not something ... that we should focus at first.

@markknol

This comment has been minimized.

Copy link
Member

markknol commented Feb 10, 2018

Awesome, Keep up the good work guys! I think this will improve Haxe a lot!

@mikicho

This comment has been minimized.

Copy link

mikicho commented Feb 10, 2018

@ncannasse what about performance? I mean, if feature X will develop in Haxe->Ocamel it will be slower than if written in Ocamel directly? the difference is negligible?

@nanjizal

This comment has been minimized.

Copy link

nanjizal commented Feb 10, 2018

It would be very interesting if there was an example of what Haxe code might look like for targeting Ocaml so a snippet showing a small aspect of functionality. I am not even sure on what support Haxe has for immutable code at the moment ( there is now final, but I have not seen it used ).
I think it would be really helpful to flush out some small examples even if incomplete, rather than just describe them above, because when we can start to see how Haxe might be used it gives more idea of current limitations. For instance to capture the input and interest of really able Haxe developers who don't use Ocaml currently. Also I would be keen if macros were used sparingly, I think they are very powerful but increase the entry point of contribution and could make code more difficult than the current Ocaml codebase.

@ncannasse

This comment has been minimized.

Copy link
Member Author

ncannasse commented Feb 11, 2018

@mikicho the idea is to have 0 loss of performance in the process, so we are looking for something almost 1-to-1 conversion

@nadako

This comment has been minimized.

Copy link
Member

nadako commented Feb 11, 2018

So I started a little ocaml project for converting ml to hx here https://github.com/nadako/ml2hx. It's really rough but the foundation is there to improve upon. It works by invoking ocaml as a lib and processing the typed AST to generate Haxe code. Currently requires 4.06 and probably won't compile with earlier versions, because the compiler API incompatibilities, but oh well.

@back2dos

This comment has been minimized.

Copy link
Member

back2dos commented Feb 12, 2018

I'm a little surprised this didn't get mentioned: https://github.com/elnabo/haxe_in_haxe ^^

Either way, this is very exciting news. I really hope something will come out of it ;)

@nanjizal

This comment has been minimized.

Copy link

nanjizal commented Feb 12, 2018

@nadako hi very interesting, obviously my first thought was to try the code on itself so I can understand the Ocaml code better:

$ ./main main.ml
File "main.ml", line 2, characters 5-13:
Error: Unbound module Asttypes

Should this work?

( By the way could the make file work without ocamlfind and use opam instead, it's not in homebrew and not needed for make haxe compiler? Well I have installed ocamlfind here, but for other mac users might be better to avoid? I don't know? )

@mikicho

This comment has been minimized.

Copy link

mikicho commented Feb 12, 2018

@nanjizal Please discuss about his awesome project in its repo.
Guys, please stay focus on the subject, we want this issue will be readable in the near future :)

@fullofcaffeine

This comment has been minimized.

Copy link

fullofcaffeine commented Feb 13, 2018

Could someone shed some light on https://github.com/elnabo/haxe_in_haxe? Is it a coincidence that the creation of this repo somehow matches the creation of this issue?

I guess the idea is to first automatically transpile the OCaml codebase to Haxe using https://github.com/nadako/ml2hx. and then fix / improve manually as we go?

That said, those are really great news! 🎉

@nadako

This comment has been minimized.

Copy link
Member

nadako commented Feb 13, 2018

Is it a coincidence that the creation of this repo somehow matches the creation of this issue?

It's not, it was published when discussions started about Haxe in Haxe :) I'm not sure about the fate of this repo though, since porting this amount of code by hand is not really an option.

I guess the idea is to first automatically transpile the OCaml codebase to Haxe using https://github.com/nadako/ml2hx. and then fix / improve manually as we go?

Yeah, that's the idea at the moment. Thanks to the fact Haxe is actually quite similar to ocaml (enums, pattern matching, var shadowing, everything as expression, etc.), it's pretty straightforward to generate readable-ish haxe code from ocaml.

@elnabo

This comment has been minimized.

Copy link
Contributor

elnabo commented Feb 14, 2018

Could someone shed some light on https://github.com/elnabo/haxe_in_haxe? Is it a coincidence that the creation of this repo somehow matches the creation of this issue?

I started to try and do a copy of the compiler in Haxe a month ago. And when the issue of writing Haxe in Haxe arose, @ibilon told me to make my code public.

I don't know how long, I'll continue it, because it's likely other options will succeed first. But for the moment, it help me understand a bit better the compiler.

@skial skial referenced this issue Feb 14, 2018

Closed

Haxe Roundup 419 #477

1 of 1 task complete
@nanjizal

This comment has been minimized.

Copy link

nanjizal commented Feb 14, 2018

@elnabo well it may turn out that for hxcpp this approach is better, we can't assume that ocaml/functional output will provide the best results, but I can imagine that an automated initial port that nadako has started should provide a solution that more easily covers all the edge cases currently. But you can never be sure where such projects leads I certainly urge you to continue sometimes working from different angles also brings a better overall solution and new opportunities. Can you compile the JS target with your code?

@elnabo

This comment has been minimized.

Copy link
Contributor

elnabo commented Feb 14, 2018

I tried to compile with hxnodejs but I get weird error Class<Sys> has no field stderr so I must have misconfigured something.

But as I try to be as similar to the source as possible. Me and nadako should end with similar looking code. Except I'm likely to have bugs hidden somewhere, as I think I currently have.

@Simn Simn added this to the Backlog milestone Apr 17, 2018

@Simn

This comment has been minimized.

Copy link
Member

Simn commented Mar 18, 2019

Rant

You're crazy

I just read through this issue properly for the first time and think y'all are crazy. Porting the Haxe sources to Haxe only to then compile them back to OCaml is insanity.

Let's look at what's bad about OCaml:

  1. It's a pain the ass to set up, especially if you're not on Linux.
  2. Its compilation time can be very slow.
  3. It does not support true parallelism.
  4. Its tooling in general is a bit shit, even if you are on Linux.

Which of these problems would be alleviated by having the Haxe sources in Haxe and compiling them to OCaml? Well, we would still be stuck with the same crappy toolkit, except that we now have another layer on top of it which makes most of the issues even worse:

  1. We still have to set up OCaml in order to compile Haxe.
  2. We still have a slow OCaml compile time, plus we now add the Haxe compile time.
  3. We could still not use parallelism because we go through OCaml.
  4. Alright, we could use some of Haxe's better tooling.

Furthermore, all that is assuming that we would even make it that far in the first place. I don't share your optimism that the code base could be converted to maintainable Haxe and be generated back to efficient OCaml. There's no way that you could generate idiomatic Haxe to efficient OCaml in general, so we would have to gimp ourselves by essentially writing OCaml-idiomatic code in Haxe.

Alternatives

So in summary, I think this entire idea of going through OCaml should be dropped. Which obviously raises the question of what we should do instead. Given that we want to write Haxe-in-Haxe, I can see two possibilities:

  1. Write a JVM-target and use that.
  2. Focus on hxcpp.

Given that 1. requires an additional step, it would seem that 2. is the best option. At this point, hxcpp is pretty battle-hardened. And guess what, it already has a generational GC, which is the reason why Nicolas clings to OCaml. I can't speak to the efficiency of it, but in my experience all it takes to get Hugh on a job is a benchmark showing a problem.

What's more, hxcpp supports parallelism. Even if its GC doesn't beat OCaml's (which I expect to be the case), I'd wager that the usage of multithreading would make up for that.

And yes I'm implying that we should rewrite Haxe instead of trying to awkwardly port it. I know people think it's not an option, but I disagree. We're currently sitting at just above 100kloc, which is not much. I can easily write you a parser + typer + optimizer + JS generator in one month of focused effort if I have a clear vision of what to go for.

I'd have to think a bit about how to port eval, but that can also be figured out. The analyzer would require some redesign that doesn't depend on functors. Porting generators requires some manpower, but there's nothing inherently difficult about it either.

IMO this is what we should go for with Haxe 5. There's a million questions still to be answered obviously, but I really want to get the point across that targeting OCaml is a huge strategic mistake in 2019.

@nadako

This comment has been minimized.

Copy link
Member

nadako commented Mar 18, 2019

I'm not sure how much different the rewrite would look like from the "OCaml-idiomatic code", especially if we talk about multi-threading. I'm pretty sure it'll still be a bunch of recursive pattern matching functions and persistent data structures, unless "we" want to go for the OOP visitor pattern madness (I can see different array handling and some proper non-exception based control flow though:)). That's why I think that automated port is technically viable, even though with all the extra punctuation that Haxe will bring, our single-letter names will look even more messy :)

Personally I have no problems with OCaml nowadays, and it's tooling is slowly getting better, but I could of course work on Haxe in Haxe. In some situations Haxe would be actually nicer (e.g. reification and/or macros for expr building).

Regarding the run-time, I was recently writing a compiler for AS3 in a functional style, simiilar to how Haxe is written, and I can say this requires a really really good GC, because even V8 (node.js) chokes badly with a lot of persistent data object allocations. I had to add quite some boilerplate checks to do less allocations so it doesn't die. HXCPP seemed to perform better, but I didn't do deeper benchmarking, but I feel like at this point "we" have a lot of experince to improve GC :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.