-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add a "Format" button in the Editor #5986
Comments
I was discussing about this issue with @angelikatyborska and @neenjaw, Tim basically teased me into creating an Elixir/Phoenix app that could do this service for "all" track languages. So I threw together a very basic proof of concept here: https://github.com/jiegillet/exercism-formatter. It works fine (for 2 hours of work put int):
So I wanted to throw the idea in the ring. Of course, there are many things to consider, such as
Happy to discuss more :) |
Surely network topology can solve this issue? I'd definitely vote for a micro-service here (assuming formatting is safe) vs spawning dockers every time a single format is needed for a single language... though it's entirely possible that a small fleet of dockers might run/host this micro-service... |
Thanks @jiegillet. I think the main two things are:
Once we've answered (1) we'll be able to move forward, so I suspect the key route is to:
Determining if something is safe is basically "Can running this execute code the user uploads". For languages that have macros as part of compilation, then the answer may be yes although it intuitively feels like it should be no. For example, lots of languages cannot safely have their compiled ASTs extracted without potentially running user-uploaded macros. Re the hosting, if they're safe, then using Lambda to do this is trivial. You just upload the docker images and they'll just insta-spawn whenever needed and then shut down again. The extra time is only 500ms and you only pay for the time things are on. The snippet-extractor does this for example. However, if they're not safe, then all this gets much more complex and you need an infrastructure like what we have for representers. |
If by |
The answer is no. There have been several CVEs across languages where specifically crafted content could break through the barrier and turn into a remote executing exploit. That said. I do want this for JS/TS! |
Ugh. This is why we can't have nice things. 😭 Cant we do it in browser for sure for JS/TS? |
Yes. |
I think the right next step is to get a couple of spikes for various languages. I think we need:
And then it's just some work in building out infrastructure for me. Thoughts? |
Btw, I'm presuming in this that formatting works with one file, rather than needing the whole project. Does this hold true across languages? |
I really like this idea. I could easily do a spike for C#/F#, they both have formatters that are fairly easy to use.
All the regular instructure restrictions should indeed apply (read-only file system, no networking). @SleeplessByte Do you perhaps have a link to one of the CVEs to get an idea of what the attack vector was?
I don't think it does. Many formatters will need another file where the formatting is specified (e.g. a |
OK, cool. So we need to pass a directory as (1), not a file? And there's going to need to be some system to get the right files in again. |
Yeah, I'd say: pass in the directory like with the other tools, but additionally also pass in a filename. |
Agreed with needing the directory. @ErikSchierboom I didn't search, but I, coincidentally, had this open, which is the same type of exploit on a static analysis tool. Note that this one was because of how the plugin was written, not the underyling tool (eslint), but it's the same thing. https://www.cve.org/CVERecord?id=CVE-2021-27081 |
I was really curious about this so I built a WebAssembly powered formatting for Go: go-fmt-wasm.netlify.app Would be great if someone could test this on Mac/iOS/Safari. It was much easier than I thought. WebAssembly (and the support some languages provide for it) really came a long way in the past years. I will share some thoughts about what I think might be the challenges for making this work in production later. |
Looks great! Works for me on Chrome and Safari on macOS. Took a while for it to work the first time on both... |
@ynfle Yes, I also had the feeling the initial wasm loading takes a while. |
Some more thoughts on the WebAssembly version:
All in all, it seems still a bit early for doing the formatting in the respective language via WebAssembly. Go has a built-in formatter that everyone is supposed to use and requires no additional configuration file so might be an easy starting point. It would be fine with "one file in, one file out". Not sure how to go about finding out whether executing the formatter is save of not though. I would like to challenge the assumptions that we need to pass in/provide a folder. Shouldn't everything besides the file to format be baked into the docker image already? Configuration files, all the packages needed etc. I can't really imagine a case where the formatting of a file would depend on other files in the folder or where the config would differ by exercise. 🤔 |
I love that you "just build it". It works very well. Initial load we can trick around, including making the button unavailable during first load. |
Just to pick up on this bit and provide context, running this via Lambda isn't possible if the tools aren't "safe" (which they won't be), as AWS reuses lambdas between sessions, so one person's code could wreck the container for the next person. So there's work in getting these running in a similar way to how test/runners/analyzers/etc work. Not huge work - but substantially more work involved than there would be if we could go down the Lambda route. What I'd probably do for this is take a simpler route than we do for submissions as we don't need persist the results etc. One thing we do need to ensure is that we don't accidentally become the "hosted formatting tools" platform for use outside of Exercism, else that could get expensive fast! So we'll need to find a way to guard against that too. The WASM thing solves all this, which is why it's exciting as an approach. So if we can go with that, then that would be a massive win. (And it would also allow us to make reusable wasm components for this, which would be a cool wider OSS contribution!) |
Ok, I didn't quite get that we already decided we need the secure test-runner like setup. If we want to try the WASM route, I would be happy to pilot the WASM powered "Format" button for Go. As DJ said, we could visually indicate when it is ready to be pressed (disable it, show a spinner in the button, something like that). |
Maybe you, @SleeplessByte, and I pair on getting that done as a project over the festive break? It'll only take us an hour or so I imagine, but could be pretty fun! :) |
Sounds like a plan! |
I took a look at the gofmt wasm demo. Very cool and seems to work well! My suspicion is that the initial page load is not really something that can be improved much using the standard Go toolchain. Programs that compile to wasm have to statically link the portions of their languages standard library that they use. That seems to be a fair bit for Go! It looks like hello world is ~2mb. This really varies based on the language. I did run Tiny Go seems to be a common means that Gophers use to generate smaller WebAssembly binaries. I'm not sure about the tradeoffs, but it might be worth taking a look to see if that can generate something functionally equivalent. |
For the wasm browser embedding environment, it's very true that there isn't a standard for what gets passed into a WebAssembly module. It's essentially ad-hoc and all projects generate code differently. WebAssembly on the server does have a standard API for all the normal systems type things called the "WebAssembly Systems Interface." This provides a sort of "syscall interface" and enables similar sorts of host capabilities as Docker or an OS process. Typically, there are different compiler targets for these things. Something like |
@bushidocodes Thanks a lot for having a look! I wasn't aware Tiny Go was related to WebAssembly at all but they even say that on the "box". 🙈 According to their docs, they support the bit that we need: I will check this out in more detail when I have some time. Also thanks for that second comment, that confirms a lot of the suspicions I had when looking into this. |
When using TinyGo as @bushidocodes suggested, the size of the wasm file drops from ~3MB to ~0,5MB. The time from initial page load to then the format button can be pressed was reduced from ~1s to ~0.5s. So this definitely improves the performance. The TinyGo version is available here: The format button is now disabled until it is ready to be pressed and I added a section where it shows the loading time. Caveat: TinyGo has a misconfiguration in their current version that leads to a stack overflow when one tries to format larger code samples. They fixed this here tinygo-org/tinygo#2447 but they did not release a new version yet and their "build the |
Nice! It looks like it's only 166kb over the network due to compression. Pretty impressive speedup! Bummer about the configuration issue! If you want to accelerate this, some tools have CLI options for stack time ( Possible Reference: tinygo-org/tinygo#2108 |
@bushidocodes Thanks again for the help! What you suggested with the custom target configuration file worked perfectly. And it turned out that the increased value they set is not enough for our purposes so the new release wouldn't have solved our problem. Instead of doubling the original stack size I tripled it and now it works just fine even for very large chunks of code (~3000 lines). I merged the tinygo version into main now so it's available under the main URL now: https://go-fmt-wasm.netlify.app/ Do you also have some input regarding proper error handling? Ideally an error should be thrown when the formatting does not work and then caught so that the website could indicate the failure to the user and report the error (e.g. to bugsnag). This is what I found how to do this: https://stackoverflow.com/a/67441946/5077542 Maybe you could have a look whether the explanation and code there make sense. |
Huh, looks like Long term, there is also a phase 3 proposal to add native exception handling support to WebAsssembly. I assume that would allow thrown errors to compile to a WebAssembly primitive that the JS embedding environment could catch in a JS try/catch block without the glue code, but I've not yet used it so I'm not sure. I suspect the browser support matrix for this is very poor, but it does seem to be in Chrome. https://github.com/WebAssembly/exception-handling |
For languages that have formatting tools (Elixir's
mix format
, Elm'selm-format
, Go'sgofmt
, etc...), especially if the formatting tool is a community standard, it would be great to have a "Format" button on the online editor.It would involve creating new repos for languages that want to support this.
The text was updated successfully, but these errors were encountered: