Proposal to split linking part of emcc.py into emlink.py

[sbc100]

Today emcc.py act as both the compiler driver and the linker. In the interests of separation of concerns and in order to make emscripten more componentized I've been working on a change to put all the linking logic in emlink.py and have emcc.py call this if needed under the hood.

This change should also allow for other use cases, such as taking an externally built wasm file (from rust, or go, or kotlin, or wasi) and running emlink.py on it in order to "emscriptenize" it and create JS and html around it.

[sbc100]

@kripken we spoke briefly about this. I've got a WIP here: https://github.com/emscripten-core/emscripten/pull/new/seperate_link

[kripken]

What do you mean by "linking logic" specifically? (is it the linking of system libraries, calling of wasm-ld, js linking, and/or post-link stuff like finalize and wasm-opt that we sometimes call the "link" stage?)

[kripken]

I forgot to say, I greatly like this idea!

This change should also allow for other use cases, such as taking an externally built wasm file (from rust, or go, or kotlin, or wasi) and running emlink.py on it in order to "emscriptenize" it and create JS and html around it.

[sbc100]

All of the above I think. With the exception perhaps of selecting which system libraries to include. I'm not sure if its better to do that in emcc.py still.

[dschuff]

I think this is a good idea in principle. It does bring up some interesting questions about where certain bits of functionality go. For example you could argue that some of the post-linking stuff could belong logically to the link step, or belongs as a separate step invoked by emcc (or perhaps in a third wrapper, if there is a use case for separating it from both the link step and from emcc).

[kripken]

If it's all of the above, then it seems like we can almost just rename emcc.py to emlink.py? And create a new emcc.py that calls that emlink.py to do most things, but also handles compiling source files, and that keeps all the existing emcc.py flags and behavior.

Is that basically the direction here? If so that sounds good!

[dschuff]

In the long run you could even imagine using clang itself as the compiler driver, and invoking custom steps post-link (and/or perhaps the link wrapper instead of lld). Then we could remove all of that from emcc. There would still be a lot of compatibility flags though, so that might be undesirable unless we also want a flag reset. But we could have emcc.py just for flags and still remove compiling-c-files logic if the behavior is close enough.

[sbc100]

If it's all of the above, then it seems like we can almost just rename emcc.py to emlink.py? And create a new emcc.py that calls that emlink.py to do most things, but also handles compiling source files, and that keeps all the existing emcc.py flags and behavior.

Is that basically the direction here? If so that sounds good!

That is an intersting idea! The actuall approach I took was to basically take the bottom half of emcc.py and move it to emlink.py (basically everything after is prints "stopping at bitcode"). It seems to be mostly working.

[kripken]

@dschuff

In the long run you could even imagine using clang itself as the compiler driver, and invoking custom steps post-link

That sounds good to me. We do have emcc --cflags which emits all the flags you need to call clang with (paths to our system libs etc.). If we make the system libs more standardly structured maybe even those can be unnecessary.

@sbc100

The actuall approach I took was to basically take the bottom half of emcc.py and move it to emlink.py (basically everything after is prints "stopping at bitcode")

I see, yeah, that sounds good too - maybe the two approaches would end up pretty much the same, actually.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 7 days. Feel free to re-open at any time if this issue is still relevant.

[kripken]

I think we still want this @sbc100 ? Odd stalebot tries to close issues with someone assigned to them.

[stale]

This issue has been automatically marked as stale because there has been no activity in the past year. It will be closed automatically if no further activity occurs in the next 30 days. Feel free to re-open at any time if this issue is still relevant.

[sbc100]

We now have --post-link which makes emcc.py act in this way.

How ever I still think it would be nice to split the linker code out of the emcc.py file.. just as an internal detail.