Espruino modules packaged as NodeJS modules #2550
Replies: 19 comments
-
Posted at 2015-03-02 by @gfwilliams I know you've come across https://github.com/espruino/espruino-tools and the code it uses - I think some of this would be really good if it was integrated into that? Specifically I'd love to get browserify (or something like it) working inside the Web IDE - ideally with some offline minification too. If it were able to drop functions that weren't used, and inline/rename ones that were (I guess this is the idea?) then it'd save a load of Espruino's RAM. I wonder whether a better way to accomplish that (than re-packaging all of Espruino's modules) is to override the 'module getter' in browserify to just look online for the Espruino modules? As chrome doesn't have access to the local filesystem, creating a copy of all the modules there won't really help - and as the library of modules grows over time it'll become slower and slower. Or were you doing this so that the editor you use can find the modules and follow function definitions into them? As far as Spire, will it use EspruinoTools? I ask because a lot of work has gone into (and will continue to go into) making EspruinoTools format code in the right way for Espruino in order to avoid subtle problems. Re-implementing that code really doesn't help very much, but if you could actually make EspruinoTools provide that nice stream interface (which the WebIDE Terminal could then connect to) it would mean that people could pull in that library, or in fact use the command-line tool and pipe to stdin/out. In the module you ask about the binary notation. It's because when accessing registers in hardware it's often a lot easier to read. It's part of ES6 so you could probably just run Just wondering about |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-02 by boneskull Let me think this stuff over and get back to you this evening. I'm thinking an |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-02 by boneskull (FWIW, I grabbed the name so someone wouldn't do something dumb with it. Simply "squatting" a module name is not allowed and can get your publishing privs revoked, so needed to have something there anyway.) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-03 by @gfwilliams Hmm, it's a tricky one... 'write Espruino code in NodeJS' is a big vague - I don't think you actually want to develop Espruino code in NodeJS. Instead, you want tools that help you to write code for (and to interface with) a device that is running Espruino? To me, that kind of means the command-line tools. You don't need the modules on your PC to develop with, because they're automatically uploaded by the command-line tools as they are needed. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-03 by alexanderbrevig I guess he's thinking in the line of writing server side code that controls espruinos in an API-like fashion, rather than simply sending it serial commands? Something like
|
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-05 by boneskull Hi, sorry didn't have time to reply until now. No, it's neither of these things really. :D Before I go into more detail, please understand that come from a software engineering background. I work developing a large suite of SaaS web apps, as well as OSS projects in my free time. I've encountered many issues centered around dependency management, code integration, and deployment. I'm not trying to tell you guys, "I know what I'm talking about". I just have, perhaps, different a experience here, and I'd like to use it to propose a few things. So, what I'm about to suggest may not have a 1:1 relationship to the hardware world. If I were to have originally designed how the Espruino modules were packaged, and toolchain structured, it would be different than what exists now. Maybe you agree with my points, but haven't had the resources to take action. Or, if you disagree, hopefully you can tell me why I'm wrong, because my experiences with software are not applicable. At the very least, it should be interesting to think about things this way! Espruino Modules in npm
Dependency management is a difficult problem. npm represents many thousands of hours of development time working to solve this problem. Out of the many package management systems that I've used, npm is likely the best. The Espruino Web IDE has all but ignored dependency management by opting for automatic installation instead. Not everyone wants this! Automatic InstallationAutomatic installation of required modules seems very strange to me. I understand that this can help people get up-and-running, here's why I find it harmful:
The only practical drawback I can think of to using explicit versioning is that code must be written to support it. Mitigating this is that 98% of the work will done by npm, and not the Espruino toolchain. The "extra step" of declaring your dependencies can be avoided (see below). A caveat is that most Espruino users probably don't care about this stuff, which means it must be implemented in such a way that it's transparent. An Alternate WorkflowTo support those that do care about versioning, like myself, I'm envisioning an alternate workflow like this (it could be an "advanced" mode or something):
Keeping What We HaveTo retain the current workflow--and make these changes transparent--but also be compatible with the alternate workflow:
"Deployment"Browserify can do basically everything necessary here, coupled with a custom transform. One thing I don't understand is this: what is the point of not minifying your files before uploading? You can't really debug it anyway, right, so why do you care what your code looks like? Regardless, I've identified the following steps:
Where and HowWe should spend a bit more time thinking about the "minimum viable product". It's clear to me that we don't have to do this all at once, but what's the bare minimum we can start with?
Why?This buys Espruino a lot.
If I have any grumbles about the Web IDE, it seems that it's gone the same path the Arduino IDE has--it does not provide an API. To use an Arduino, you basically need to be a human interacting with a window. There is a CLI tool, ino, which is pretty unstable, but also seems the best alternative. It's unstable because the Arduino IDE team will freely introduce breaking changes--why shouldn't they? It's not an API! Likewise, the Web IDE is not an API. Anything package trying to leverage it will face the same instability, unless you happen to develop both the Web IDE and a CLI-tool--because you know what's going to break. But the Web IDE shouldn't be an API, it should be a layer on top of one. Granted, it's much nicer than the Arduino IDE, but still should just be an interface. In conclusion... Ideally, if the resources and interest is there, I'd like to work with Espruino instead of in parallel. I don't want to develop and maintain an entire separate toolchain on my own. I would probably just continue to patch things together so it works for me, but I have no interest in maintaining "competing" tools. Let me know what you think. These are just a bunch of ideas. I'd be happy to discuss all of them. And I apologize if they've been suggested before and shot down--I just got this board a couple weeks ago. :D |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-05 by alexanderbrevig Some thoughts in random order: Espruino IDE vs 'API': NPM: Also, making npm available could quickly lead to a lot of questions about why things does not work etc. The Espruino is an embedded device with low memory and the interpreter does have some 'quirks' which a module author should know about and leverage/work around. Pulling in something 'simple' as _underscore will not probably be very successful. "I make a small change to my code": In conclusion: Maybe the Espruino could be an npm module? I don't mean to seem overly negative, I would probably like this feature - but I'm not convinced it's the right approach. My all too many bytes on the matter. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-05 by @gfwilliams Ok, I'm a bit busy right now, so forgive me if I'm missing something...
You can debug by checking and setting variables, and changing code on the fly. A lot of Espruino users do this routinely using the left-hand side of the IDE. It's actually one of the main ideas behind Espruino. If it wasn't I'd have just got the Web IDE to compile code on the PC, and Espruino would have been much faster than it currently is. However as it is, you can come back to a project, plug it in to your PC, and tweak it without even having the original source code available. You're coming at this from the point of view of a node.js developer and assuming it's way behind Tessel because it's not what you're used to. But the reality is that they are aimed at very different areas... Tessel is going after node.js devs and that's great - but Espruino is about creating things quickly and easily without having to learn a bunch of extra tools.
Libraries changing under you is a tricky one, and yes, it is extremely frustrating. NPM does solve that, however any solution to the problem needs to run completely in the Web Browser with extremely limited permissions. Right now you can actually program an Espruino board from your iPhone's headphone jack, from a normal website, without any special apps or software. I'm not convinced that what you're actually suggesting will do that? If you're suggesting that everyone needs to have filesystem access, install node, npm, and a bunch of modules first then it's not an improvement at all.
Please don't do that, at least try and leverage EspruinoTools. I can pretty much guarantee that you'll lose interest in a few months, and then it'll just be sitting there on NPM along with node-espruino, grunt-espruino and espruino-cli as yet another tool that uploads 'hello world' but fails when faced with anything more complex. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-05 by @gfwilliams Sorry, I don't want to seem negative - but it seems to me that maybe you haven't used Espruino that much yet, and you're trying to apply the lessons you learned on million-line enterprise projects to a device that can handle maybe 10k lines maximum? It's not to say they're not good ideas - but the reality is that NPM for Node.js is a necessity because of the amount of different libraries people tend to pull into their projects. On Espruino we haven't even hit the stage where version control is a minor annoyance yet. At the moment, I think <10 people have contributed modules to Espruino. Keeping the modules together means I can be a lot more sure that they're of good quality and all work with Espruino, and I think that's a real bonus when compared to something like Arduino and Node.js where you have 5 different libraries which all do the same thing. At some point we'll reach the stage where I can't keep track of everything, and then yes, NPM will make a huge amount of sense. But until then I think switching over to NPM may not actually be in everyone's best interests. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-05 by alexanderbrevig
There's this http://www.espruino.com/Reference#l__global_edit :) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-05 by HerrLinder Hello, |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-05 by @allObjects I can see both worlds... and kind-of 'going-back' to more 'free-flying' after or while still working with one leg+ in a 'perfectly well', very heavy (prison-gang)-tool-chained world is a challenge... to say the least... as for me. That does not mean that we have to be stupid and throw out all what 'we' have learned since the mythical man month and while growing with all the more recent code dev, build, test, deploy, and management 'techniques'. But applying them in their spirit by living out a more self-disciplined approach than in a system-over-policed way is the way in this resource-constrained (Espruino) micro controller world. I feel like being allowed to take the trainings-wheels off when switching to the Espruino platform. ...may be it is because I learned (and still learning) in the well guarded context. On the other hand I have to confess that even with Espruino I still take advantage of the guarded world:
As you - @boneskull - can see, there are things going on into your direction, but it should - as you say - be transparent to the user - if the user does 'not care' explicitly - and should happen in the spirit of Espruino... without compromising its resource frugalness. For sure, more is to be conversed about and figured out how to make available for the Espruino community. Btw, @AlexanderBrevig, thanks to point out function edit. It does not only allow to modify (particular) code in an instance on the living object (without loosing the data) but also capture the final version to put it into the (edited) file... ;-) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-06 by boneskull
I'm not sure I understand what you're getting at here.
No, this will not work on an iPhone or whatever. You would need NodeJS installed.
I want to first point out I'm not saying Espruino is "behind" Tessel. Please forgive me if I gave that impression. I understand they are different tools. Perhaps what I didn't understand is that "ease of use" is paramount to the project. With the "alternate" workflow I detailed, you should probably know a little more about what you're doing, but regardless, the interaction with npm would probably be obscured behind the IDE. Maybe you missed the bit about "keeping what we have" which would not require the user to even know npm was behind the scenes. But, it sounds like, at the current time, using npm is a no-go because we don't want to rely upon a filesystem? If the likelihood of a user programming an Espruino from an iPad is greater than the likelihood of a user wanting npm integration, so be it. I have no data in either direction.
Please consider this constructive criticism: I began writing my own CLI tool because leveraging EspruinoTools is extremely awkward. It's not an npm module, so I can't add it as a dependency. It's not versioned, so even if I did, it'd be a crapshoot. It doesn't seem to expose an isolated API layer, which means you need packages like Ideally, it would become the "first-class" API for communication with the Espruino, and the Web IDE itself would simply consume it. But without some sort of buy-in, I don't see the project as having much future. I think I mentioned (In a perfect world, the Arduino IDE and I will relinquish the Gordon, I would like to apologize for coming out of left-field on this, and trying to jam the Espruino into a hole which it clearly does not fit. As I wrote, my point of view is as a software engineer. Hardware is a hobby of mine. I'm interested in increasing accessibility of hardware to software developers, so they can more readily apply their skills to the platform(s). Hardware which supports high-level languages is a great start; I suppose that's what drew me to Espruino originally. Another way is to support common toolchains and methodologies. There are a lot of things. (I do sense a general unease about the influx of software developers to hardware, perhaps spurred by the popularity of the Arduino. I am concerned I won't get much support in my efforts. Some of this venom comes from garden-variety elitists who don't like the "dumbing down" of hardware, but others arguments include a lot of "ain't-broke-don't-fix-it". For example, I'd smile if the official Espruino modules each had unit tests. Funny, right? Maybe to some it seems like extra complexity, or simply unnecessary. When you're hacking on a breadboard, you don't have a second breadboard that asserts the first breadboard works--you just look at it, and if it works, it works. I don't feel that the same strategy works well for software libraries, even if those libraries run on a microcontroller.) Anyway, I don't need to spend another 2 hours writing this post... :D |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-06 by alexanderbrevig @boneskull What scenario do you want to realize with an API? Then
*This is written from memory on my windows disc, the code is on linux so may be something wrong. Let me know if you want me to post the real stuff... |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-06 by @gfwilliams One of the reasons for Espruino's creation was the initial 'ease of use' - putting all the tools I could on the chip itself meant that you need very little in the way of software to communicate - in fact just a serial terminal for simple coding. It's unfortunate that with modules, Assembler, minification, etc. the software on the host has actually become quite a bit more complicated - but at least it can still be installed with a few clicks in Chrome... As a node dev you'll already have node.js installed and will be used to it, so it's easy to just assume it'll be there. I think it'd surprise you how much difficulty people would get in installing it though - especially somewhere like a school where the IT department is very strict about what can and cannot go onto the system. There's also the issue of platform support - while communication from iPad/etc isn't really used at the moment (with blockly it might get some use in education soon though), a surprising amount of people do use the Web IDE with Chrome OS. That's not to say we can't use NPM though - there's already some simple support in the Web IDE for NPM modules containing a single file (you mentioned gzip being included, and it's there for unpacking those modules). It could be expanded relatively easily, although obviously it's a shame to reimplement something that already exists (it doesn't have any of the version control stuff at the moment). As I'd said before I don't think we should move all the modules over to NPM just yet, but at least it doesn't rule it out. The tests are definitely a good idea, and then having a separate repo per module starts to make a lot of sense. While modules that just interface to hardware can't have many meaningful tests, we're starting to get non-hardware libraries like MQTT that would benefit hugely from testing. I totally agree about EspruinoTools - it really needs to be a module that can easily be used. The way I see it, there are a few separate parts: CommunicationsThis is really bad at the moment, with several JS files implementing the same interface. You just have to load the one you want. Ideally we'd have something that could load multiple communication interfaces. You'd call Now here, streams would seem like a good bet. I guess you know more about this than me, but are they something that is implemented on the browser, or are they Node-specific? If they're node-specific I'd guess it's actually a no-go again, but we can just implement Code transformationAs I guess you'd noticed, there's sort of a framework where each bit of code places itself inside a global 'Espruino' variable, uses a 'Config' object, and calls Basically I had to come up with something that'd run in the browser with no filesystem access, and I wanted to use the code directly (rather than shoving it through browserify) so it could be debugged and changed from within Chrome's debug tools. UI stuffThe UI is handled the same way - the Web IDE just adds to what EspruinoTools provides. It's not fantastic, but it's there now, and it works on node and in the browser.
This sounds like a good plan. Personally, rather than lifting code out of EspruinoTools, could we not just add an API to the existing EspruinoTools repo and stick it in NPM? It wouldn't be an epic refactor, but would mean that EspruinoTools became something truly useful. The existing command-line tool could then be modified to use that API and added to NPM too, and everyone's happy? At some point later on we could refactor what's there, but as you say, at least there would be a consistent API so even if the internals were changed, it didn't matter. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-06 by boneskull @gfwilliams Actually it seems NodeJS and ChromeOS are not mutually exclusive. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-09 by @gfwilliams Yes, you can - but it looks like you have to put it into developer mode first (which IIRC wipes the hard disk). Sadly it's not as easy as going to the chrome web store and clicking 'install' :) |
Beta Was this translation helpful? Give feedback.
-
Posted at 2017-05-18 by trusktr Webpack's shiny new feature is tree shaking (when writing with ES6 Module format rather than CommonJS, or using a webpack loader that takes current CommonJS and outputs ES6). We can set Babel as a transpiler, and make a babel-preset-espruino which compiles only the language features that Espruino doesn't have natively yet. After that, running the "webpack" command can compile a minified bundle from an entry point (unused code not included, and duplicate code consolidated), then one more step with the espruino CLI will upload the file. I'm sure that can then be easily automated in the IDE. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2017-05-22 by @gfwilliams Will webpack run fully in-browser? Definitely having some tree-shaking, plus the ability to minify using some of the ES6 that Espruino supports, would be awesome. |
Beta Was this translation helpful? Give feedback.
-
Posted at 2015-03-02 by boneskull
Hey, I'm working on a little project to integrate the Espruino a bit more tightly into common NodeJS toolchains.
First, I'm repackaging the modules into something NodeJS can understand. That's underway here (
master
branch only).I have another project, Spire, which is, at first glance, another one of those "command-line" tools to upload scripts. But, beneath the surface, there's a Duplex stream--theoretically you should be able to consume it as you would any other writable or readable stream. Its current implementation communicates via a serial port and STDOUT, but I would like to create an abstraction layer over this, so you can simply stream data to/from the Espruino via NodeJS. For example, you could set the Espruino to serve real-time data over a socket, then use NodeJS to pipe it into a database, file, or web app. There are other ways to accomplish this sort of thing, of course, but when dealing with large amounts of data, streams allow you to avoid memory consumption--this is especially important if you wish to send large amounts of data to the Espruino.
Anyway, I'm mostly just playing with ideas at this point, but I'm wondering if anyone has any input or would be interested in the outcome.
I think the Web IDE could benefit from using a communication interface such as this--and as-of-yet unwritten apps that people may wish to build on top of it.
Beta Was this translation helpful? Give feedback.
All reactions