Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New magic %%executable to compile current AST #316

Merged
merged 2 commits into from Mar 31, 2020
Merged

New magic %%executable to compile current AST #316

merged 2 commits into from Mar 31, 2020

Conversation

hahnjo
Copy link
Contributor

@hahnjo hahnjo commented Mar 19, 2020

The cell content is used as the main function. After declaration, use a RecursiveASTVisitor to generate LLVM IR for all currently parsed FunctionDecls and VarDecls at file scope in the AST. Then emit object code to a temporary file and finally invoke the linker to produce a binary that can be execute with the ! magic command.

@hahnjo
Copy link
Contributor Author

hahnjo commented Mar 19, 2020

This very likely needs user documentation in docs/source/magics.rst, but I first want to check if you would be in favor of including a feature like this. We have found a ton of use cases, starting from using ThreadSanitizer (which would be very tricky in a JIT context) over more control in a benchmark environment to possibly using MPI.

I have a commit locally that hooks up things needed for -fsanitize=thread. Currently it just checks if the argument is part of the linker arguments (string comparison, so no real parsing), but I can include it if you want.

CMakeLists.txt Show resolved Hide resolved
@SylvainCorlay
Copy link
Member

SylvainCorlay commented Mar 21, 2020

Thanks @hahnjo and sorry for the late reply!

I would be very happy to include this. Do you want to include some documentation before we move forward and merge this?

@hahnjo
Copy link
Contributor Author

hahnjo commented Mar 21, 2020

Thanks @hahnjo and sorry for the late reply!

I would be very happy to include this. Do you want to include some documentation before we move forward and merge this?

Great! Yes, I'll add some user documentation on Monday.

@hahnjo
Copy link
Contributor Author

hahnjo commented Mar 23, 2020

@SylvainCorlay PTAL

@SylvainCorlay
Copy link
Member

SylvainCorlay commented Mar 23, 2020

@hahnjo I am curious about the behavior of the resulting binary when you make use of functions from the kernel (to display things in the notebook etc).

What would be really awesome would be for the binary to actually link with xeus 😄

@hahnjo
Copy link
Contributor Author

hahnjo commented Mar 23, 2020

@hahnjo I am curious about the behavior of the resulting binary when you make use of functions from the kernel (to display things in the notebook etc).

What would be really awesome would be for the binary to actually link with xeus smile

Not sure what the expected result would be: You're running the binary as a new process and Jupyter just forwards the output streams. So there is no (direct) connection to the notebook that you could use for communication.

@SylvainCorlay
Copy link
Member

SylvainCorlay commented Mar 23, 2020

More generally, can the result be linking with other binaries?

@SylvainCorlay
Copy link
Member

SylvainCorlay commented Mar 23, 2020

Where I am going with this - which is kindof a moon shot and probably out of scope for this PR is the folllowing:

We have this project called "Voilà" which turns notebooks into standalone web apps. In the case of the C++ kernel, since the content of the notebook is fixed, there is no real need for the C++ to be "interpreted", and we could actually turn the "notebook content + kernel" into a single executable that speaks to the voilà front-end. By all practical purposes, the resulting executable would be "a kernel" that only executes the content of the notebook and responds to comms.

@hahnjo
Copy link
Contributor Author

hahnjo commented Mar 23, 2020

More generally, can the result be linking with other binaries?

Yes, that is actually the reason for passing the additional arguments of %%executable to the linker. I've successfully used this for OpenMP (-fopenmp), TSan (-fsanitize=thread; needs some tweaks for the CodeGenOptions), and MPI (-lmpi).

@SylvainCorlay
Copy link
Member

SylvainCorlay commented Mar 23, 2020

cc @JohanMabille

@hahnjo
Copy link
Contributor Author

hahnjo commented Mar 25, 2020

I've rebased this PR on top of current master and also reformatted if statements (sorry for not noticing earlier). The new version also explicitly handles FunctionDecls and global VarDecls instead of relying on HandleTopLevelDecl to ignore "invalid" decls: The old code triggered assertions when enabled.

Jonas Hahnfeld added 2 commits Mar 25, 2020
The cell content is used as the main function. After declaration,
use a RecursiveASTVisitor to generate LLVM IR for all currently
parsed FunctionDecls and VarDecls at file scope in the AST. Then
emit object code to a temporary file and finally invoke the linker
to produce a binary that can be execute with the '!' magic command.
@hahnjo
Copy link
Contributor Author

hahnjo commented Mar 31, 2020

@SylvainCorlay @JohanMabille any update on merging this? I've also cleaned up my implementation of rudimentary option parsing for -fsanitize=thread, but I think this should be a separate PR...

@SylvainCorlay SylvainCorlay merged commit 04ba489 into jupyter-xeus:master Mar 31, 2020
2 checks passed
@SylvainCorlay
Copy link
Member

SylvainCorlay commented Mar 31, 2020

Sorry I actually thought that I had merged. This would have gotten into the next release anyways :)

Many thanks for this!

@hahnjo hahnjo deleted the dump-executable branch Mar 31, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants