-
Notifications
You must be signed in to change notification settings - Fork 610
Building Toasters #273
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Comments
... I hope I'm not talking like a crazy person here.
Not more than most of us here (or at least me :-P ). You have shown a
really interesting use case that was between the initial ones when
@groundwater started the project and also between mines when I got a ride,
only that it hasn't got enough traction to work toward that idea :-)
At this moment, NodeOS architecture is designed so you can custom it by
forking the project and changuing the dependencies of the layers to the
ones you want, in your case it would a custom nodeos-initramfs that instead
of executing nodeos-mount-filesystems it would exec your app, and remove
nodeos-rootfs and nodeos-usersfs ones. This way you could get a pure
initram-based and diskless system. Problem is that your app would be
running as root... that's not fine. A better alternative would be instead
use a custom nodeos-usersfs with just a non-root user that exec your app,
but this would require a disk partition and it's not yet documented, but
has the advantage of execute it as regular user and allow data persistence.
Your idea of a builder gor the initram is not bad and can be really useful,
but nothing is done towards this. You could be able to create a toaster
tool in an independent npm module that generate an initramfs and then use
the just generated one to build your custom copy of NodeOS. In that way, it
would be ideal to first split nodeos-mount-filesystems to an independent
module with the basic system config, and later exec your app with
restricted permission. How do you think about doing it? :-)
|
I kind of want to do it, but I've got other stuff to do too. And right now I just need to get something working. Anyways, I don't see any problem with running as root, since there isn't going to be anything else on the system anyway. No point trying to mess around with permissions—the only thing they could do is restrict access to low number ports, which needs to be done on the firewall anyway. The only hardware that would be connected to the system would be hardware that my program would need. Making a user account just doesn't make sense. I might try poking around a bit, but I can't commit a lot of time to this. |
Then in your case the quick path is to fork nodeos-initramfs and replace And just curious... are you planning to use NodeOS on production? Where?
|
Well, it's a bit early to say if this will be put into production, but I'm looking to use it for running some internal automation where I work. Since you already make pre-built images for the project, would it be possible to have |
I would be really happy to hear that ;-)
Yes, each layer is independent, so you can be able to use nodeos-barebones |
Hi @sbuller, did you get any chance to take a look on this, or was |
I'm still poking around. I feel like I'm getting close to having something that does what I need. |
I have described a possible solution at NodeOS/nodeos-initramfs#4, tell me if we have some points in common here. |
@sbuller, did you get any progress on this? |
I've gotten a bit sidetracked. I put together nos-init to generate initramfs files in the manner I envision. I've played a bunch with syslinux and other bootloaders, as well as related filesystems. I can't say that it's truly worthwhile, but I had been focused on generating a bootable disk image without depending on tools external to NodeJS/NPM. I settled on syslinux with fat16 is the most realistic option. It took me a while to come to the realization that there was no practical way to produce the output entirely with streams. I've got a bunch of spaghetti code at this point that puts together a lot of this but it's super hairy, and it's still missing some important parts. The stupid thing is that it's probably just a couple lines of bash script using mkisofs. I'd just rather make program that could run everywhere. That's one of the main things I like about NodeJS. Anyhow, it's still pinging around my brain at the moment, but hasn't seen any work for a week or two. |
That's awesome! :-D A generator of
That would be cool if it helps to make NodeOS build environment more portable, although by far I have been seeing that the most portable language at least for low-level and platform independent components is bash. Look, it now has official suport on Windows 10, too... :-P
Is that the reason why you are doing the A bootable NodeOS image on a FAT pen drive booting directly to a secure environment, that's directly the USB drive root directory, that would be interesting... OverlayFS would have problems with that, but maybe we could be able to override them with FUSE or an upgraded version of UMSDOS... Hum... :-) |
My intent is only to have the initramfs and barebones on the fat16 image. My use case is only for qemu. That said, I do try to keep things modular and general where I can. I can't remember my reasons for abandoning symlinking, but it's not because of fat16. The initramfs is still a cpio which populates a tmpfs. |
Seems you are creating a /boot partition, that's just the purposse of
In that case, QEmu can directly load and run the linux kernel and the
Yes, I read the code, I like your approach :-) |
I actually really like the idea of using qemu to directly load the kernel and initrd, but using a disk image fits better with my current setup. Using disk images I can run my NodeOS images in the same way as I run all my other images. |
Ok, that totally makes sense :-) How are you creating the NodeOS images? El 9/8/2016 17:18, "sbuller" notifications@github.com escribió:
|
Well the overall idea is to create a fat16 image containing A fat16 image has 4 main parts:
The annoying thing about this is it basically needs to be written back to front, which also makes it difficult to use space efficiently. I tried https://github.com/owenson/tiny-linux-bootloader, but couldn't get it booting at all. |
How are you creating the FAT16 image? Are you using the genfatfs module... or have you created a FAT16 formater in Javascript?!? O_O
Maybe you could not be able to use the Streams API, but you could be able to create first the data section and later add to the front the filesystem structures...
Really interesting, specially the part of having the kernel and the initrd separated from the partition, this would be really useful to create single-user bootable pendrives hidding the kernel and the initrd files from the user when used as a regular USB drive :-) How far did you get with that? Could be the problem that it uses a By the way, have you written more extensively how you have done this on a blog or something? I would be interested on read it on detail :-) |
Just about ...
Some of these aren't hard to fix, but I'm not planning on working on them. |
So yes, you are doing a (limited) FAT16 formatter in Javascript... :-D It's a shame can't be done streaming but makes sense, usually filesystems structures are at beginning. I don't know if the root filesystem could be put at the end...
I find the limitations acceptable for that use case. Are you leaving zeroes in the fields that you are not initializing so others can fill the holes later? It would be a good alternative to the usage of |
I'm about to pack it in for the day, but you can have a look at what I'm doing: |
I have been taking it a look, it's neat :-D |
Well, it's looking like the filesystem writing is working. It looks like a couple variables need to be changed in the bootsector code when it's written though. |
Couldn't them be calculated in advance? El 10/8/2016 21:13, "sbuller" notifications@github.com escribió:
|
I'm not sure. I'm looking into 3x 4-byte discrepancies in the boot code at offsets 0x11A, 0x120 and 0x12F. 0xDEADBEEF and 0xFEEDFACE definitely look like placeholders. I may be able to simply hardcode some values, but I'm not sure what I'm looking at. I'd managed to boot an image I put together by hand last week, so I have some hope. |
That offsets are inside the actual boot code, so I doubt they are other thing that instructions in machine language. That values can be both placeholders or guard marks, so if it works without that changes, I'll left as is. What are the actual values you see different both before and after? |
It turns out ldlinux.sys also needs some processing. I'm starting to second guess this approach. I must have been playing with already processed files and didn't manage to deviate far enough from the parameters. Maybe. I don't know. I've been reading through the syslinux source; for reference the relevant code seems to be the function syslinux_patch() in the file libinstaller/syslxmod.c. Trouble is, even if I work out all the modifications I need to make, what happens with the next release of syslinux? And here I thought generating the filesystem from scratch would be the hard part. This will probably have to stew in my head for a while. I'll keep reading syslinux source for now, but I don't see a way to get syslinux working quickly. At least the fat16 code is theoretically useful even as it is. |
I'm not sure what problem do you have with SysLinux, but could it be that you are generating a partition image instead of a disk image and SysLinux can't be able to manage that in a cleanly way? |
No, if I run the syslinux program on the fs/disk image that I generate it works fine. The issue is that I've been trying to put the files syslinux uses on the fs myself, rather than letting syslinux do it. It turns out syslinux doesn't just copy a couple files and write a generic boot sector—it does those things, but then makes a few changes. Anyhow, it looks at this point like the tiny-linux-bootloader is a much more attractive option. |
I use the installer, so can't help you :-( Maybe you would exec it in a second step, or ask SysLinux guys about what changes are they doing...
Lol! X-D At least it's an interesting one... :-) |
This is awesome, keep up the good work guys =) I've had this idea that I wanna build NodeOS docker images that just runs my apps, this way I can just have my CI server run a build step on git pushes and deploy the generated docker images to a private docker image registry and then I have a registry of versions of my app as docker images available to deploy from =) |
Well, you are in good news, because due to popular demand we have been adapting NodeOS to make it possible to use as a single-process OS, and now it's possible to build custom initramfs images running the modules in an isolated environment. It needs some testing though, but probably is what you are looking for your Docker CI system. Your system seems to be very interesting, if you need some help to adapt NodeOS to it or to integrate it in your system I can offer you some contractor advice, so if you are interested don't doubt on send me an email :-) |
With the new structure on RC3 it would be trivial to make |
As a potential user, I'm finding the focus of the documentation and tooling a bit different than what I'd hope for. I think my aims are still in line with the direction of the project though.
My hope is to have a tool—let's call it
toast
for arguments sake—that builds a bootable disk image when run in a node repository (i.e. a directory containing a package.json file). I don't particularly care if it's built from source—in fact, I'd probably rather thattoast
would download a pre-built image. The downloaded image should be combined with the contents of the repository such that the resultant image can be booted by qemu, and the js file pointed to by "main" in the package.json file should then be started automatically.I specifically do not care about users, or any other direct form of interaction or administration. My goal in terms of administration is that when I want to change things, I modify the source code in my package, re-run
toast
generating a new image, and upload it—entirely replacing the old image. In view of this, I would recommend a read-only filesystem.Beyond this, some facility or API for mounting additional disks would be nice, but if that proves insurmountable then I would rather store my state elsewhere on the network rather than store my state alongside my application. Perhaps arguments passed on the command line to
toast
could be stored in the image as well, and be referenced by the running script, allowing an application to be more easily instanced.Example
main.js
package.json
Command Line
... I hope I'm not talking like a crazy person here.
The text was updated successfully, but these errors were encountered: