Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add an embeddable execution mode #248

Open
nex3 opened this Issue Mar 11, 2018 · 13 comments

Comments

Projects
None yet
4 participants
@nex3
Copy link
Contributor

nex3 commented Mar 11, 2018

It's desirable to be able to use Dart Sass from various different language environments, with functions and importers defined in the host language, while still getting the language semantics of Dart Sass and the performance of the Dart VM. We could theoretically expose the VM plus Dart Sass as a C API, but figuring out how to get that to build reliably (let alone efficiently) seems like a nightmare.

A much easier solution would be to create an executable that uses stdin/stdout to allow a caller to define functions and importers. This would require defining a protocol of some sort—probably either JSON- or protobuf-based—for communication and for serializing Sass values.

This should probably be in a separate package from Dart Sass proper, but that package doesn't exist yet so I'm filing this here for now.

@xzyfer

This comment has been minimized.

Copy link

xzyfer commented Mar 12, 2018

We've also considered the latter in Node Sass as an alternative to doing native binding. Would be interested to see where this discussion goes, and if it can be general enough for other language implementations.

@nex3

This comment has been minimized.

Copy link
Contributor Author

nex3 commented Mar 12, 2018

We should definitely coordinate to come up with a good, cross-implementation protocol.

@nex3

This comment has been minimized.

Copy link
Contributor Author

nex3 commented Jun 15, 2018

Upon consideration, I don't expect that I'll personally have time to create a Ruby or JS wrapper using this. It's still something I think would be valuable, but I want to wait to start in on it in earnest until we have someone signed up to write at least one wrapper that makes use of it.

@nex3

This comment has been minimized.

Copy link
Contributor Author

nex3 commented Jan 18, 2019

@strychnide has volunteered to create/maintain the Node side of this, so we can start thinking about it more seriously. I think the first question is, what technology should we use? The two main criteria as I see them are:

  • Portability. We want to make it straightforward for any language to embed, which probably means using well-defined protocols that have existing libraries in many languages. We also want to make sure it's possible for LibSass to eventually expose the same API to aid its own embedability.

  • Efficiency. The whole point of embedding the Dart VM rather than using the JS compiled version is for its increased speed versus V8. We should aim to lose as little of this speed as possible in overhead, even for stylesheets that make many native function or importer calls.

I think protocol buffers as the transport method is a good choice for both of these. Protobufs are much more compact and quicker to de/serialize than JSON (the other obvious option), and they have client libraries for Dart, Node.js, and Ruby.

At a higher level, we also need a protocol for defining the calls that the embedder can make into the embedded executable. This is trickier than most RPC systems, because we want RPCs to flow both ways. We need the embedder to be able to ask the executable to compile files, and we need the executable to be able to ask the embedder to run functions and importers. I think we have two options here:

  1. Use gRPC. The big benefit here is that this is totally standard, and again has existing libraries for Dart, Node.js, and Ruby. However, it makes the strong assumption that RPCs are made in one direction to a remote server. It requires HTTP/2, which I suspect (but have not verified) is slower than communicating directly over stdin/stdout; and it doesn't have explicit support for callbacks. We could fake them with bidirectional streaming, but at that point we'd be halfway to rolling our own protocol, since we couldn't model and server-to-client requests as proper RPCs.

  2. Roll our own. This would require more specification work on our part as well as more work for users to generate, but it would let us communicate directly over stdin/stdout and we could easily support callbacks. We could also potentially base this on existing specs that don't use protobufs; for example, JSON-RPC 2 supports bidirectional calls over a generic byte stream.

My intuition is that option 2 is best, but I'm open to disagreement (or another alternative I haven't considered).

@strychnide

This comment has been minimized.

Copy link

strychnide commented Feb 2, 2019

Using gRPC sounds like a massive overshoot to me, HTTP/2 is not meant for this kind of implementations (and gRPC is very microservice oriented imho).

I'd be in favor of protobufs, maintaining the definitions in an external library, then wrapping it in Dart and making it a dependency (I'd wrap it in node and make it a dependency too, so that we could sync the changes).

I'm just providing input, I don't really see a negative side in json-rpc either, I just think that protobufs are more standard and require less work to maintain, but feel free to dispute this point (I don't have that much experience and you'll be the one deciding/implementing it first, so choose what you think works best).

if you're unsure just implement a small part of the functionality in a "testing" branch both with json-rpc and protobufs, I'll wrap them in node so that we could have something to base our assumptions on.

TL;DR option 2 seems better

EDIT: it came to my mind later: do we really need bidirectional calls? From my point of view the wrapper just starts dart-sass with the correct options and then doesn't send anything afterwards, it just receives and parses output. Am I missing something?

Sorry for the late reply, been busy studying :)

@nex3

This comment has been minimized.

Copy link
Contributor Author

nex3 commented Feb 6, 2019

I'm just providing input, I don't really see a negative side in json-rpc either, I just think that protobufs are more standard and require less work to maintain, but feel free to dispute this point (I don't have that much experience and you'll be the one deciding/implementing it first, so choose what you think works best).

The big downside is the parsing and serialization overhead. The protocol may end up being quite chatty when used to implement functions in the host language.

EDIT: it came to my mind later: do we really need bidirectional calls? From my point of view the wrapper just starts dart-sass with the correct options and then doesn't send anything afterwards, it just receives and parses output. Am I missing something?

The wrapper needs to be able to define functions and importers in the host language that can be called by the Sass implementation. That's where most of the complexity of the protocol will come from.

TL;DR option 2 seems better

I agree. Let's plan on Protobufs + a custom RPC protocol then. I'll create a repo for the protocol definition in the next day or two.

@nex3

This comment has been minimized.

Copy link
Contributor Author

nex3 commented Feb 11, 2019

Creating a new repo involves some bureaucracy on the Google end, so it may take a bit, but I should have it up soon.

@strychnide

This comment has been minimized.

Copy link

strychnide commented Feb 22, 2019

Any progress?

@nex3

This comment has been minimized.

Copy link
Contributor Author

nex3 commented Feb 22, 2019

I'm waiting on one last approval to create the repo.

@nschonni

This comment has been minimized.

Copy link
Contributor

nschonni commented Feb 23, 2019

There is this https://github.com/Microsoft/language-server-protocol which is used by VS Code for multiple languages and remote debugging between IDEs

@nex3

This comment has been minimized.

Copy link
Contributor Author

nex3 commented Feb 25, 2019

Okay, I've created https://github.com/sass/sass-embedded-protocol and sent you an invite to a team with write access. I'll send out a pull request soon with an initial draft of a protocol skeleton.

@strychnide

This comment has been minimized.

Copy link

strychnide commented Feb 27, 2019

@nschonni since I feel bad leaving a suggestion without an explicit answer, the point (IMHO) is that we're not trying to communicate with an IDE and we currently don't know what the final spec is going to look like (although Natalie knows more than me for sure), so we may need additional informations (we'll probably do) and/or not use the entirety of the spec (which, to my limited understanding, is just a set of field in a JSON-RPC implementation).

This is a very simple answer, I know, it's hardly accurate and surely I'm missing many things (I gave the spec only a superficial look) so feel free to point out any mistake in my answer.

@nschonni

This comment has been minimized.

Copy link
Contributor

nschonni commented Feb 27, 2019

no worries, I was just sharing some prior art that might have some overlap

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.