Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Timeline for integrating into Rust upstream? #67

Open
davystrong opened this issue Dec 1, 2023 · 19 comments
Open

Timeline for integrating into Rust upstream? #67

davystrong opened this issue Dec 1, 2023 · 19 comments

Comments

@davystrong
Copy link

Hi,

I'm wondering if there's any timeline for this project to be integrated into the upstream Rust compiler? As I understand it, the Rust compiler needs to be modified somewhat to emit the information this project needs. Are these changes too specific to ever get merged? It would be great if this could be installed as a package simply using cargo, instead of having to compile the whole compiler.

Sorry if there is information about this somewhere, I couldn't find it!

@ZuseZ4
Copy link
Member

ZuseZ4 commented Dec 1, 2023

Hi, the plan is to directly merge this work into nightly rustc, it also got an accepted Major Change Proposal in the past, even though rustc dev still need to vote on the final PR.
There are some implementation changes I need to finish, but I got this fork down to only add 2.5k extra lines of logic and the requested changes are comparably small, so I hope to get to it in December.

@ZuseZ4
Copy link
Member

ZuseZ4 commented Dec 1, 2023

No worries for asking and thanks for your interest, most of the discussions happen on the rust-lang zulip, either in the autodiff stream, or in a help stream linked in the autodiff discussion.

@davystrong
Copy link
Author

Hi @ZuseZ4, thanks for that, that's really helpful!

@OwenTrokeBillard
Copy link

I, too, am looking forward to Enzyme being merged into Rust. Thanks for all your excellent work so far @ZuseZ4!

@OwenTrokeBillard
Copy link

@ZuseZ4 Is there a checklist of issues blocking a merge with upstream Rust? Any issues you could use help with?

@ZuseZ4
Copy link
Member

ZuseZ4 commented Jan 3, 2024

Hi @OwenTrokeBillard, thanks for getting back to this. I was on vacation around end of year and previously got stuck for a while on some re-design, but recently managed to re-implement a core part of my autodiff macro in a way that is more liked by other rustc devs. I now just need to port the remaining code of rust-enzyme to the latest branch, since rustc is somewhat fast moving, so there is a little bit more refactoring required. I think it's the easiest if I just finish this on my own.

One place where I would appreciate help though is the website and documentation, hosted at https://enzyme.mit.edu/rust.
It is developed from https://github.com/EnzymeAD/rustbook
In case you know any other AD tools whose documentation you like (e.g. jaxbook, enzyme.mit.edu/julia, Zygote, ...), it would be great if you could try to add missing pieces to our documentation. Jed is also interested, although slightly blocked since he can't test any examples on the latest branch, and needs to use a slightly dated version. In general, also just adding testcases (and finding bugs) would be great.

As an alternative, it would be great to have rust-enzyme in our compiler explorer: https://enzyme.mit.edu/explorer
I am not really great when it comes to webdev / CI, and the rust side is slightly harder than the other language frontends, since in our case we need to compile all of rustc for it to work.

@jedbrown
Copy link

jedbrown commented Jan 3, 2024

It looks like explorer will just need a build script similar to this and then a flag to activate it. https://github.com/EnzymeAD/enzyme-explorer/blob/main/builder/build-enzyme.sh
A good task once the rebase is ready, in no small part since it makes it easier to report issues.

@OwenTrokeBillard
Copy link

Documentation and compiler explorer support are important. I'll gladly help where I can. However, those don't appear to block upstreaming. Do they?

It would be great to have a checklist of issues preventing an upstream specifically. Based on other PRs in Rust, it looks like the process can take many months. Better to get the ball rolling early.

@OwenTrokeBillard
Copy link

For example, I imagine Windows support is required (#54). I will give that another try soon.

@ZuseZ4
Copy link
Member

ZuseZ4 commented Jan 5, 2024

Windows support should only be fixing the path issues, I'll try to unblock you here after lunch.

Other than that it's the new macro built-in design instead of our old approach which was requested by other rustc devs, since we would otherwise break cross-compilation.

Some rustc devs already gave their ok from their point of expertise, the final approval will be from t-lang though and they have quite limited time. Someone will bring up the autodiff macro at their next meeting once we are ready, but we should have some good docs by then so they can easily grasp what we would introduce on the language level (that is our autodiff macro frontend). The middle and backend are less relevant here, since we can change those without affecting users.

@fknfilewalker
Copy link

fknfilewalker commented Feb 9, 2024

Is enzyme now part of nightly? I saw the issue was closed but can not get it to work.

@ZuseZ4
Copy link
Member

ZuseZ4 commented Feb 10, 2024

Are you referring to this one rust-lang/compiler-team#611 @fknfilewalker?
That Major Change Proposal (MCP) was accepted, but there is still some implementation work left.
Mainly this fork needs to build and tested under more targets (MSVC under Windows especially),
and in some more configurations (e.g. downloaded LLVM from CI instead of self-build LLVM, in Docker, ...).
Happy to accept help if you have some time, but otherwise can't give a precise timeline on when it will be ready.

@fknfilewalker
Copy link

I am not really a rust user, but I thought about switching when I read that EnzymeAD should be in upstream. I was just confused weather it is or not. So rust is not directly one of my priorities, sorry.

@jmelo11
Copy link

jmelo11 commented Apr 14, 2024

Hi, where can i find information about the status/installation/usage of this integration?

Thanks in advance

@ZuseZ4
Copy link
Member

ZuseZ4 commented Apr 14, 2024

The installation instructions are in our readme: https://github.com/EnzymeAD/rust
Usage examples are on our website: https://enzyme.mit.edu/index.fcgi/rust/usage/usage.html
The current status is that I'm waiting for llvm/llvm-project#83052 to get merged, afterwards for rustc to move to an llvm with this patch integrated, and finally I will propose to upstream this fork.

@jmelo11
Copy link

jmelo11 commented Apr 15, 2024

Thank you so much!!

@ZuseZ4
Copy link
Member

ZuseZ4 commented May 1, 2024

@ all. The Rust language Team met this morning and decided to approve this as an experiment within rustc!
rust-lang#124509

So we can now actually upstream this. I will start the refactoring in the next days. There is still the unmerged llvm PR which we want to have in, but in the best case I can hopefully push my upstream PR at the end of this month.

@jmelo11
Copy link

jmelo11 commented Jun 16, 2024

@ all. The Rust language Team met this morning and decided to approve this as an experiment within rustc! rust-lang#124509

So we can now actually upstream this. I will start the refactoring in the next days. There is still the unmerged llvm PR which we want to have in, but in the best case I can hopefully push my upstream PR at the end of this month.

Hi @ZuseZ4, does this mean that it can be installed via nightly or is this still in progress?

@ZuseZ4
Copy link
Member

ZuseZ4 commented Sep 6, 2024

The first 13 files were upstreamed last night: rust-lang#129176
The remaining ~70 files will probably take a few more weeks to be reviewed and cleaned up. Once everything is upstreamed, I'll flip the build flag to ship them on nightly without people having to build the official rustc themself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants