Feature request: Rust and Cargo

[ghost]

Hi!

Would it be possible to include Rust and Cargo in a-shell? As far as I know Rust has a lot of interoperability with C/C++, so I have no idea if it would be possible to include in a future version.

Thanks in advance!

[oflebbe]

Hi I have exactly the same question, that's the reason I am exploring the build process of a-shell.
Are there any deep dependencies of a-shell to python?
I would rather like explore to have a-shell only to have the shell commands, clang, vim and cargo/ rust targeted to WASM . (i.e. remove python and tex ), leaving in clang and adding rust and its supporting commands.
(python, tex and rust communities are not that much overlapping).

Any recommendations where to start ?

[holzschu]

Hi, these are very good questions. a-Shell does not have any deep dependency to python, python is just treated like one of the many commands available.

To compile a command, you will need to:

• download the ios_system.framework (use the binary release at https://github.com/holzschu/ios_system)
• compile the command for the iOS architecture: CC = clang, CFLAGS="-arch arm64 -miphoneos-version-min=14.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.3.sdk, LDFLAGS is like CFLAGS, plus -F ...Frameworks -framework ios_system.
• (once it compiles) make the system create a dynamic library rather than an executable
• embed that dynamic library inside the executable ("Embed Frameworks", in "Build Phases")
• add the command to Resources/commandDictionary.plist. It's an array of arrays, indexed by the name of the command, with 4 entries for each command:
  • name of dynamic library
  • function to be called inside the library (usually main)
  • options (the string that is passed to get_options, can be left empty)
  • whether the command operates on files, directories or not (file for a compiler).
• (once all this is done) start the command and observe what happens, debug any crash.
• (once it is done) edit the source files for:
  • output to stdout/stderr should be replaced by output to thread_stout/thread_stderr
  • reading from stdin should be replaced by reading from thread_stdin
  • make sure all memory is released when leaving the program
  • make sure all variables are initialized when we restart the program
  • (these two are related to a fundamental issue: we never actually exit from a program on iOS, so the system will never reclaim the memory we allocated and reinitialize the variables. It is up to us to do the cleaning).
  • lines using std::cout will have to be replaced by fprintf(thread_stdout, ...), I haven't yet found a way to redirect C++ standard streams.
• (once all this is done): for AppStore release, convert the dynamic library into a framework (see examples in https://github.com/holzschu/python-aux/)
• (once all this is done): submit a pull request, if you want the command to become part of the main application. I see your idea about having a third, separate app for a-Shell, targetting rust more than python and TeX.

The first step can be difficult if the package or command insists on guessing the compiler and flags rather than letting you set them.

If your command is simple and fast (so, not rust or cargo) you can also compile it to webAssembly, which makes things a lot easier.

If the command uses fork and exec, you will need to edit the source code a bit more, as fork() is a no-op in iOS. iOS_system has a fake fork, but we go through both branches in the same thread.

[oflebbe]

Thanks for the detailled instructions.

But... rust is a llvm based compiler, cargo it's packaging and build system :=) Challenge is to support a different compiler. My proposal for a seperate app is not to enlarge the bulky and ressource heavy a-shell by adding the resource intensive rust compiler on top. If I understood correctly, the App Group Feature would make it possible to interact a "rust shell" with "a-Shell" , right?

[holzschu]

You probably want to have a look at https://github.com/holzschu/llvm-project, then. But compilers are just like other commands, only larger. For a compiler to create something that can be executed, it has to produce webAssembly, which should be feasible for rust.

[ghost]

Hi, these are very good questions. a-Shell does not have any deep dependency to python, python is just treated like one of the many commands available.

To compile a command, you will need to:

• download the ios_system.framework (use the binary release at https://github.com/holzschu/ios_system)

• compile the command for the iOS architecture: CC = clang, CFLAGS="-arch arm64 -miphoneos-version-min=14.0 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS14.3.sdk, LDFLAGS is like CFLAGS, plus -F ...Frameworks -framework ios_system.

• (once it compiles) make the system create a dynamic library rather than an executable

• embed that dynamic library inside the executable ("Embed Frameworks", in "Build Phases")

• add the command to Resources/commandDictionary.plist. It's an array of arrays, indexed by the name of the command, with 4 entries for each command:

  • name of dynamic library
  • function to be called inside the library (usually main)
  • options (the string that is passed to get_options, can be left empty)
  • whether the command operates on files, directories or not (file for a compiler).

• (once all this is done) start the command and observe what happens, debug any crash.

• (once it is done) edit the source files for:

  • output to stdout/stderr should be replaced by output to thread_stout/thread_stderr
  • reading from stdin should be replaced by reading from thread_stdin
  • make sure all memory is released when leaving the program
  • make sure all variables are initialized when we restart the program
  • (these two are related to a fundamental issue: we never actually exit from a program on iOS, so the system will never reclaim the memory we allocated and reinitialize the variables. It is up to us to do the cleaning).
  • lines using std::cout will have to be replaced by fprintf(thread_stdout, ...), I haven't yet found a way to redirect C++ standard streams.

• (once all this is done): for AppStore release, convert the dynamic library into a framework (see examples in https://github.com/holzschu/python-aux/)

• (once all this is done): submit a pull request, if you want the command to become part of the main application. I see your idea about having a third, separate app for a-Shell, targetting rust more than python and TeX.

The first step can be difficult if the package or command insists on guessing the compiler and flags rather than letting you set them.

If your command is simple and fast (so, not rust or cargo) you can also compile it to webAssembly, which makes things a lot easier.

If the command uses fork and exec, you will need to edit the source code a bit more, as fork() is a no-op in iOS. iOS_system has a fake fork, but we go through both branches in the same thread.

That’s very interesting, and I think I’ll have a look at that just for the learning experience. I think it might become handy down the road in my studies!

I know there’s already a fair share of rust compatability with wasm, so I guess it should be somewhat possible to have something working. No idea about cargo though.

[holzschu]

I should probably have added a step 0 for compilers: first compile the compiler on your usual computer, and check that it can produce webAssembly programs that are compatible with the wasi-sdk: https://github.com/holzschu/wasi-sdk and https://github.com/holzschu/wasi-libc.
(with this modified wasi-libc, you get access to local files).

[ghost]

I’ll look into this! Thank you so much, it’s very interesting and nice to get some more hands-on experience

[oflebbe]

rust target wasm32-wasi works out of the box. That's really great.

[NobodyXu]

Thanks for the detailled instructions.

But... rust is a llvm based compiler, cargo it's packaging and build system :=) Challenge is to support a different compiler. My proposal for a seperate app is not to enlarge the bulky and ressource heavy a-shell by adding the resource intensive rust compiler on top. If I understood correctly, the App Group Feature would make it possible to interact a "rust shell" with "a-Shell" , right?

AFAIK, a-Shell already support clang and it is gcc that is not supported by a-Shell.

So maybe it’s not that hard to support cargo, but you probably need to configure cargo to use existing llvm toolchain dynamically instead of compiling another one.

[NobodyXu]

IMHO, one of the problem for supporting rust is that it uses custom lang_start function (called by _start) that initialize argv, heap, unwinding, backtraces and etc, so instead of calling main, one would need to invoke _start direcrly.

It seems that lang_start is actually called in main, so that shouldn’t be any problem at all.

[personalizedrefrigerator]

This comment about compiling miri (and I think also rustc) to WASM is interesting: rust-lang/miri#722 (comment)

[NobodyXu]

Fun fact:

cargo can actually can be used as a library.

[NobodyXu]

This comment about compiling miri (and I think also rustc) to WASM is interesting: rust-lang/miri#722 (comment)

wasmer v2.1.0 can now compile wasm to native dynamic library.

[shikukuya]

pip3 install nb-cli

I have this problem too

[rquinn2]

You probably want to have a look at https://github.com/holzschu/llvm-project, then. But compilers are just like other commands, only larger. For a compiler to create something that can be executed, it has to produce webAssembly, which should be feasible for rust.

Replying here because it's the most relevant place regarding adding LLVM based languages. .I want to know if this seems possible to you?

I've been trying to get Julia in a-shell. There was some work compiling (an older version) to wasm , but it's pretty limited if you want to use any real functionality.

• Compiling Julia natively would never be allowed, because it uses a JIT which as far as I know is a non-starter for Apple.

• Julia is LLVM based, though, and we have a version of LLVM running. The other option then is to compile Julia minus LLVM natively. And use it with our existing version of LLVM (well, Julia requires some patches to LLVM that the upstream hasn't fixed, but maybe those can be added).There are problems with this too, there's a long list of dependencies. Not sure which would pose issues, but certainly seems more feasible. It does compile to Arm64 just fine on my Mac M1.

[holzschu]

I honestly have no idea. It's worth trying, though.
I would suggest starting with merging https://github.com/holzschu/llvm-project with the Julia patches and compiling. Also start with side-loading (placing the modified a-Shell with Julia on your own devices). Apple allows JIT with side-loading, just not for distribution on the AppStore... for now.

[NobodyXu]

@rquinn2 Think this might be relevant https://github.com/Keno/julia-wasm ?

[rquinn2]

@rquinn2 Think this might be relevant https://github.com/Keno/julia-wasm ?

Well, yes and no. As I understand it, the version of julia they got running there is quite limited and compiling everything to WASM is a daunting task. Additionally, having everything in WASM adds another layer of emulation to things slowing it down.

It's probably more realistic to take an approach like I mentioned above and go "half and half" - we have an LLVM here that targets WASM. So hopefully it's possible to build the rest of Julia natively to a static binary, which is nice because getting of all the dependencies built for WASM looks nasty. The Julia runtime and compiler are also able to be seperated now, if that helps us in any way.

I don't have much time to try it, but I've taken a few shots so far and haven't gotten very far. The first goal would be to get a fully native julia running in a-Shell, which as @holzschu has mentioned would not be approved by Apple (but would at least us sign a dev copy ourselves and use it). From there we'd try to use the WASM capabilities here to export the offending bits and make it "Apple safe"...if that's at all possible.

Another option is to accept the performance hit and try to use JuliaInterpreter rather than Julia's JIT.