You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Pre-#130, we didn't support print in user HDL since we use stdin and stdout for communication between processes. #130 hack-patches around that by adding a non-ASCII magic header byte to indicate the start of a protobuf message, so ASCII stdout (which is expected to be the common use case) can still be forwarded to the console. It'll probably work well enough in practice, but isn't robust to any non-ASCII input. It also adds handling complexity, since stdout needs to be 'de-interleaved' on the handling side, and this makes the stream handling different front stderr (which can be directly forwarded natively) while also adding buffering / synchronization gates.
The problem is that there doesn't seem to be a great solution.
What stdin/stdout gives us:
Over sockets, which is the other common IPC mechanism: no network access, no use of ports
Works everywhere, is part of the Python and Java standard libraries, good support
Single channel that is created between host process and subprocess, it just works
But we only get the stdin, stdout, and stderr streams, and both output streams (stdout and stderr) have existing uses that need to be deinterleaved
Potential alternative: sockets
Implementation-wise this isn't too difficult, since sockets are stream-like and there is good protobuf support for sockets
Problem is they take up OS resources (ports), though it may be possible to request an anonymous port or something
Also requires network access
Potential alternative: pipes, named pipes
Main problem is this isn't cross-platform. Windows is introducing support for Unix-style pipes, but that's Windows 10 or later. Unclear if this is supported in Python and Java without us needing to write OS-specific code.
Python allows accessing file descriptors other than stdin/out/err, but there is no corresponding support in pure (portable) Java. Hacks may be possible depending on the specific JVM used. Otherwise, this would be a great solution, open up some more file descriptors for those streams.
Potential alternative: ZMQ
ZMQ supports inter-process and intra-process communication with a sockets-like interface (send/recv) but abstracts away the transport layers. It supports several transports including TCP (network sockets) and pipes (for some OSes - apparently including support for Windows 10 after it introduced Unix-style pipes).
Both Java bindings do not seem to be maintained. JeroMQ (pure Java implementation) only supports TCP, so we might as well just use sockets. JZMQ (probably would?) support IPC, but relies on native bindings which are not available on Maven for Windows and has not seen a Maven release for about a decade.
Overall this could function as a platform-independent way to open pipes.
This adds one more pip dependency. We don't need to worry about Scala dependencies since sbt manages them automatically and we have precompiled fat JARs.
Implications for IDE development
Currently, PythonInterface exposes a stdout stream and stderr stream, which the IDE checks after each block is compiled and dumps it to the console
If we moved the proto data to a side channel, we could use more standard mechanisms to dump stdout/stderr to the console
Or not so standard, since the compilation process outlive the HdlInterfaceServer. We may have to use bits and pieces of the CommandLineState. It seems to trace all the way down to ProcessStreamsSynchronizer, don't completely understand the full stack yet.
The text was updated successfully, but these errors were encountered:
Pre-#130, we didn't support print in user HDL since we use stdin and stdout for communication between processes. #130 hack-patches around that by adding a non-ASCII magic header byte to indicate the start of a protobuf message, so ASCII stdout (which is expected to be the common use case) can still be forwarded to the console. It'll probably work well enough in practice, but isn't robust to any non-ASCII input. It also adds handling complexity, since stdout needs to be 'de-interleaved' on the handling side, and this makes the stream handling different front stderr (which can be directly forwarded natively) while also adding buffering / synchronization gates.
The problem is that there doesn't seem to be a great solution.
What stdin/stdout gives us:
Potential alternative: sockets
Potential alternative: pipes, named pipes
Potential alternative: ZMQ
Implications for IDE development
The text was updated successfully, but these errors were encountered: