Building Toasters

[sbuller]

As a potential user, I'm finding the focus of the documentation and tooling a bit different than what I'd hope for. I think my aims are still in line with the direction of the project though.

My hope is to have a tool—let's call it toast for arguments sake—that builds a bootable disk image when run in a node repository (i.e. a directory containing a package.json file). I don't particularly care if it's built from source—in fact, I'd probably rather that toast would download a pre-built image. The downloaded image should be combined with the contents of the repository such that the resultant image can be booted by qemu, and the js file pointed to by "main" in the package.json file should then be started automatically.

I specifically do not care about users, or any other direct form of interaction or administration. My goal in terms of administration is that when I want to change things, I modify the source code in my package, re-run toast generating a new image, and upload it—entirely replacing the old image. In view of this, I would recommend a read-only filesystem.

Beyond this, some facility or API for mounting additional disks would be nice, but if that proves insurmountable then I would rather store my state elsewhere on the network rather than store my state alongside my application. Perhaps arguments passed on the command line to toast could be stored in the image as well, and be referenced by the running script, allowing an application to be more easily instanced.

Example

main.js

var http = require('http');

var server = http.createServer(function (request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.end("Hello World\n");
});

server.listen(process.argv[2]);

package.json

{
  "main": "index.js"
}

Command Line

$ toast -o hello.img --ip=192.168.0.10/24 --args="80"
$ qemu -net ... hello.img

... I hope I'm not talking like a crazy person here.

[piranna]

... I hope I'm not talking like a crazy person here.

Not more than most of us here (or at least me :-P ). You have shown a really interesting use case that was between the initial ones when @groundwater started the project and also between mines when I got a ride, only that it hasn't got enough traction to work toward that idea :-) At this moment, NodeOS architecture is designed so you can custom it by forking the project and changuing the dependencies of the layers to the ones you want, in your case it would a custom nodeos-initramfs that instead of executing nodeos-mount-filesystems it would exec your app, and remove nodeos-rootfs and nodeos-usersfs ones. This way you could get a pure initram-based and diskless system. Problem is that your app would be running as root... that's not fine. A better alternative would be instead use a custom nodeos-usersfs with just a non-root user that exec your app, but this would require a disk partition and it's not yet documented, but has the advantage of execute it as regular user and allow data persistence. Your idea of a builder gor the initram is not bad and can be really useful, but nothing is done towards this. You could be able to create a toaster tool in an independent npm module that generate an initramfs and then use the just generated one to build your custom copy of NodeOS. In that way, it would be ideal to first split nodeos-mount-filesystems to an independent module with the basic system config, and later exec your app with restricted permission. How do you think about doing it? :-)

[sbuller]

I kind of want to do it, but I've got other stuff to do too. And right now I just need to get something working. Anyways, I don't see any problem with running as root, since there isn't going to be anything else on the system anyway. No point trying to mess around with permissions—the only thing they could do is restrict access to low number ports, which needs to be done on the firewall anyway. The only hardware that would be connected to the system would be hardware that my program would need. Making a user account just doesn't make sense.

I might try poking around a bit, but I can't commit a lot of time to this.

[piranna]

Then in your case the quick path is to fork nodeos-initramfs and replace
there the nodeos-mount-filesystems dependency by your app and change the
creation of the /sbin/init symlink to point to your app. After that, just
the tipical dance of npm install & npm run build & npm start. Could you be
able to do a pull-request on nodeos-initramfs with the steps that you've
followed to customize it?

And just curious... are you planning to use NodeOS on production? Where?
Can be able to tell it in a "projects and companies using NodeOS" section?
:-)
El 9/7/2016 5:03, "sbuller" notifications@github.com escribió:

I kind of want to do it, but I've got other stuff to do too. And right now
I just need to get something working. Anyways, I don't see any problem with
running as root, since there isn't going to be anything else on the system
anyway. No point trying to mess around with permissions—the only thing they
could do is restrict access to low number ports, which needs to be done on
the firewall anyway. The only hardware that would be connected to the
system would be hardware that my program would need. Making a user account
just doesn't make sense.

I might try poking around a bit, but I can't commit a lot of time to this.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#273 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAgfvoscz8vFaKfLvLHZYwcfQMH-Jj6-ks5qTw-dgaJpZM4JIbq3
.

[sbuller]

Well, it's a bit early to say if this will be put into production, but I'm looking to use it for running some internal automation where I work.

Since you already make pre-built images for the project, would it be possible to have barebones provided on its own? It looks like that will be all that I need to make me some toast.

[piranna]

Well, it's a bit early to say if this will be put into production, but
I'm looking to use it for running some internal automation where I work.

I would be really happy to hear that ;-)

Since you already make pre-built images for the project, would it be
possible to have barebones provided on its own? It looks like that will be
all that I need to make me some toast.

Yes, each layer is independent, so you can be able to use nodeos-barebones
standalone, although I recomend you to do your changes on nodeos-initramfa.

[piranna]

Hi @sbuller, did you get any chance to take a look on this, or was nodeos-barebones useful to you? If you have any question don't doubt to ask.

[sbuller]

I'm still poking around. I feel like I'm getting close to having something that does what I need.

[piranna]

I'm still poking around. I feel like I'm getting close to having something that does what I need.

I have described a possible solution at NodeOS/nodeos-initramfs#4, tell me if we have some points in common here.

[piranna]

@sbuller, did you get any progress on this?

[sbuller]

I've gotten a bit sidetracked. I put together nos-init to generate initramfs files in the manner I envision. I've played a bunch with syslinux and other bootloaders, as well as related filesystems. I can't say that it's truly worthwhile, but I had been focused on generating a bootable disk image without depending on tools external to NodeJS/NPM. I settled on syslinux with fat16 is the most realistic option. It took me a while to come to the realization that there was no practical way to produce the output entirely with streams. I've got a bunch of spaghetti code at this point that puts together a lot of this but it's super hairy, and it's still missing some important parts. The stupid thing is that it's probably just a couple lines of bash script using mkisofs. I'd just rather make program that could run everywhere. That's one of the main things I like about NodeJS.

Anyhow, it's still pinging around my brain at the moment, but hasn't seen any work for a week or two.

[piranna]

I've gotten a bit sidetracked. I put together nos-init to generate initramfs files in the manner I envision.

That's awesome! :-D A generator of nodeos-initramfs custom images, Very neat... :-) We could spand it later to make modules run inside an unpriviledged environment, but that's definitely a good starting point, good job! ;-)

I've played a bunch with syslinux and other bootloaders, as well as related filesystems. I can't say that it's truly worthwhile, but I had been focused on generating a bootable disk image without depending on tools external to NodeJS/NPM.

That would be cool if it helps to make NodeOS build environment more portable, although by far I have been seeing that the most portable language at least for low-level and platform independent components is bash. Look, it now has official suport on Windows 10, too... :-P

I settled on syslinux with fat16 is the most realistic option.

Is that the reason why you are doing the require() magic on your code? To evade problems with symlinks on FAT16?

A bootable NodeOS image on a FAT pen drive booting directly to a secure environment, that's directly the USB drive root directory, that would be interesting... OverlayFS would have problems with that, but maybe we could be able to override them with FUSE or an upgraded version of UMSDOS... Hum... :-)

[sbuller]

My intent is only to have the initramfs and barebones on the fat16 image. My use case is only for qemu. That said, I do try to keep things modular and general where I can. I can't remember my reasons for abandoning symlinking, but it's not because of fat16. The initramfs is still a cpio which populates a tmpfs.

[piranna]

My intent is only to have the initramfs and barebones on the fat16 image.

Seems you are creating a /boot partition, that's just the purposse of
nodeos-rootfs, but will be removed after 1.0 and integrated in the general
build process of NodeOS.

My use case is only for qemu.

In that case, QEmu can directly load and run the linux kernel and the
initram as parameters so there's no need to create a FAT16 partition image
to host them, in case of doubt...

That said, I do try to keep things modular and general where I can. I
can't remember my reasons for abandoning symlinking, but it's not because
of fat16. The initramfs is still a cpio which populates a tmpfs.

Yes, I read the code, I like your approach :-)

[sbuller]

I actually really like the idea of using qemu to directly load the kernel and initrd, but using a disk image fits better with my current setup. Using disk images I can run my NodeOS images in the same way as I run all my other images.

[piranna]

Ok, that totally makes sense :-) How are you creating the NodeOS images?
Maybe we can improve our build process too...

El 9/8/2016 17:18, "sbuller" notifications@github.com escribió:

I actually really like the idea of using qemu to directly load the kernel
and initrd, but using a disk image fits better with my current setup. Using
disk images I can run my NodeOS images in the same way as I run all my
other images.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#273 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAgfvuZHZqKHtNf9P-hIxsfvPRwTWy7iks5qeJongaJpZM4JIbq3
.

[sbuller]

Well the overall idea is to create a fat16 image containing barebones and the initramfs files, and make it bootable using syslinux. That's basically it. To do it without depending on external programs like mkfs.vfat and syslinux (or mkisofs which would be an alternative) involves downloading a syslinux file and extracting ldlinux.{bss,sys,c32}. These are then combined into a fat16 image. There's still a couple tricks I've got to figure out with that last part, but I've worked all the basics out.

A fat16 image has 4 main parts:

1. The bootsector - ldlinux.bss forms the bulk of this.
2. The fat - really simple, but annoying structure. Requires information about where the files are.
3. The root directory. Requires information about where the files are.
4. File data. Where to align these is the only question.

The annoying thing about this is it basically needs to be written back to front, which also makes it difficult to use space efficiently. I tried https://github.com/owenson/tiny-linux-bootloader, but couldn't get it booting at all.

[piranna]

These are then combined into a fat16 image

How are you creating the FAT16 image? Are you using the genfatfs module... or have you created a FAT16 formater in Javascript?!? O_O

The annoying thing about this is it basically needs to be written back to front, which also makes it difficult to use space efficiently

Maybe you could not be able to use the Streams API, but you could be able to create first the data section and later add to the front the filesystem structures...

I tried https://github.com/owenson/tiny-linux-bootloader, but couldn't get it booting at all.

Really interesting, specially the part of having the kernel and the initrd separated from the partition, this would be really useful to create single-user bootable pendrives hidding the kernel and the initrd files from the user when used as a regular USB drive :-) How far did you get with that? Could be the problem that it uses a initrd file instead of a initramfs?

By the way, have you written more extensively how you have done this on a blog or something? I would be interested on read it on detail :-)

[sbuller]

or have you created a FAT16 formater in Javascript?!? O_O

Just about ...
The approach I'm taking makes several assumptions to keep it simple:

1. no subdirectories
2. filesystem size will not exceed 512 * 64 * 65525 [sic] bytes + fs data.
3. no long filenames
4. no special attributes
5. no dates and times
6. all files to be added will come from existing file descriptors
7. ... more?

Some of these aren't hard to fix, but I'm not planning on working on them.

[piranna]

Just about ...

So yes, you are doing a (limited) FAT16 formatter in Javascript... :-D It's a shame can't be done streaming but makes sense, usually filesystems structures are at beginning. I don't know if the root filesystem could be put at the end...

Some of these aren't hard to fix, but I'm not planning on working on them.

I find the limitations acceptable for that use case. Are you leaving zeroes in the fields that you are not initializing so others can fill the holes later? It would be a good alternative to the usage of genfatfs, and also runtime.js guys would be interested on it... :-)

[sbuller]

I'm about to pack it in for the day, but you can have a look at what I'm doing:

https://github.com/sbuller/nos-mkfat

[piranna]

I have been taking it a look, it's neat :-D

[sbuller]

Well, it's looking like the filesystem writing is working. It looks like a couple variables need to be changed in the bootsector code when it's written though.

[piranna]

Couldn't them be calculated in advance?

El 10/8/2016 21:13, "sbuller" notifications@github.com escribió:

Well, it's looking like the filesystem writing is working. It looks like a
couple variables need to be changed in the bootsector code when it's
written though.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#273 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAgfvmqv3X8YK_9X2sy0RCroYfWO7LPQks5qeiLzgaJpZM4JIbq3
.

[sbuller]

I'm not sure. I'm looking into 3x 4-byte discrepancies in the boot code at offsets 0x11A, 0x120 and 0x12F. 0xDEADBEEF and 0xFEEDFACE definitely look like placeholders. I may be able to simply hardcode some values, but I'm not sure what I'm looking at. I'd managed to boot an image I put together by hand last week, so I have some hope.

[piranna]

That offsets are inside the actual boot code, so I doubt they are other thing that instructions in machine language. That values can be both placeholders or guard marks, so if it works without that changes, I'll left as is. What are the actual values you see different both before and after?

[sbuller]

It turns out ldlinux.sys also needs some processing. I'm starting to second guess this approach. I must have been playing with already processed files and didn't manage to deviate far enough from the parameters. Maybe. I don't know.

I've been reading through the syslinux source; for reference the relevant code seems to be the function syslinux_patch() in the file libinstaller/syslxmod.c. Trouble is, even if I work out all the modifications I need to make, what happens with the next release of syslinux? And here I thought generating the filesystem from scratch would be the hard part.

This will probably have to stew in my head for a while. I'll keep reading syslinux source for now, but I don't see a way to get syslinux working quickly. At least the fat16 code is theoretically useful even as it is.

[piranna]

I'm not sure what problem do you have with SysLinux, but could it be that you are generating a partition image instead of a disk image and SysLinux can't be able to manage that in a cleanly way?

[sbuller]

No, if I run the syslinux program on the fs/disk image that I generate it works fine. The issue is that I've been trying to put the files syslinux uses on the fs myself, rather than letting syslinux do it. It turns out syslinux doesn't just copy a couple files and write a generic boot sector—it does those things, but then makes a few changes. Anyhow, it looks at this point like the tiny-linux-bootloader is a much more attractive option.

[piranna]

The issue is that I've been trying to put the files syslinux uses on the fs myself, rather than letting syslinux do it. It turns out syslinux doesn't just copy a couple files and write a generic boot sector—it does those things, but then makes a few changes

I use the installer, so can't help you :-( Maybe you would exec it in a second step, or ask SysLinux guys about what changes are they doing...

Anyhow, it looks at this point like the tiny-linux-bootloader is a much more attractive option.

Lol! X-D At least it's an interesting one... :-)

[zimme]

This is awesome, keep up the good work guys =)

I've had this idea that I wanna build NodeOS docker images that just runs my apps, this way I can just have my CI server run a build step on git pushes and deploy the generated docker images to a private docker image registry and then I have a registry of versions of my app as docker images available to deploy from =)

[piranna]

I've had this idea that I wanna build NodeOS docker images that just runs my apps, this way I can just have my CI server run a build step on git pushes and deploy the generated docker images to a private docker image registry and then I have a registry of versions of my app as docker images available to deploy from =)

Well, you are in good news, because due to popular demand we have been adapting NodeOS to make it possible to use as a single-process OS, and now it's possible to build custom initramfs images running the modules in an isolated environment. It needs some testing though, but probably is what you are looking for your Docker CI system.

Your system seems to be very interesting, if you need some help to adapt NodeOS to it or to integrate it in your system I can offer you some contractor advice, so if you are interested don't doubt on send me an email :-)

[piranna]

With the new structure on RC3 it would be trivial to make nodeos-initramfs module to build a system like this, do you want to take a look? :-)