Skip to content

Building Toasters #273

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
sbuller opened this issue Jul 8, 2016 · 32 comments
Open

Building Toasters #273

sbuller opened this issue Jul 8, 2016 · 32 comments

Comments

@sbuller
Copy link

sbuller commented Jul 8, 2016

As a potential user, I'm finding the focus of the documentation and tooling a bit different than what I'd hope for. I think my aims are still in line with the direction of the project though.

My hope is to have a tool—let's call it toast for arguments sake—that builds a bootable disk image when run in a node repository (i.e. a directory containing a package.json file). I don't particularly care if it's built from source—in fact, I'd probably rather that toast would download a pre-built image. The downloaded image should be combined with the contents of the repository such that the resultant image can be booted by qemu, and the js file pointed to by "main" in the package.json file should then be started automatically.

I specifically do not care about users, or any other direct form of interaction or administration. My goal in terms of administration is that when I want to change things, I modify the source code in my package, re-run toast generating a new image, and upload it—entirely replacing the old image. In view of this, I would recommend a read-only filesystem.

Beyond this, some facility or API for mounting additional disks would be nice, but if that proves insurmountable then I would rather store my state elsewhere on the network rather than store my state alongside my application. Perhaps arguments passed on the command line to toast could be stored in the image as well, and be referenced by the running script, allowing an application to be more easily instanced.

Example

main.js

var http = require('http');

var server = http.createServer(function (request, response) {
  response.writeHead(200, {"Content-Type": "text/plain"});
  response.end("Hello World\n");
});

server.listen(process.argv[2]);

package.json

{
  "main": "index.js"
}

Command Line

$ toast -o hello.img --ip=192.168.0.10/24 --args="80"
$ qemu -net ... hello.img

... I hope I'm not talking like a crazy person here.

@piranna
Copy link
Member

piranna commented Jul 9, 2016 via email

@sbuller
Copy link
Author

sbuller commented Jul 9, 2016

I kind of want to do it, but I've got other stuff to do too. And right now I just need to get something working. Anyways, I don't see any problem with running as root, since there isn't going to be anything else on the system anyway. No point trying to mess around with permissions—the only thing they could do is restrict access to low number ports, which needs to be done on the firewall anyway. The only hardware that would be connected to the system would be hardware that my program would need. Making a user account just doesn't make sense.

I might try poking around a bit, but I can't commit a lot of time to this.

@piranna
Copy link
Member

piranna commented Jul 9, 2016

Then in your case the quick path is to fork nodeos-initramfs and replace
there the nodeos-mount-filesystems dependency by your app and change the
creation of the /sbin/init symlink to point to your app. After that, just
the tipical dance of npm install & npm run build & npm start. Could you be
able to do a pull-request on nodeos-initramfs with the steps that you've
followed to customize it?

And just curious... are you planning to use NodeOS on production? Where?
Can be able to tell it in a "projects and companies using NodeOS" section?
:-)
El 9/7/2016 5:03, "sbuller" notifications@github.com escribió:

I kind of want to do it, but I've got other stuff to do too. And right now
I just need to get something working. Anyways, I don't see any problem with
running as root, since there isn't going to be anything else on the system
anyway. No point trying to mess around with permissions—the only thing they
could do is restrict access to low number ports, which needs to be done on
the firewall anyway. The only hardware that would be connected to the
system would be hardware that my program would need. Making a user account
just doesn't make sense.

I might try poking around a bit, but I can't commit a lot of time to this.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#273 (comment), or mute
the thread
https://github.com/notifications/unsubscribe/AAgfvoscz8vFaKfLvLHZYwcfQMH-Jj6-ks5qTw-dgaJpZM4JIbq3
.

@sbuller
Copy link
Author

sbuller commented Jul 11, 2016

Well, it's a bit early to say if this will be put into production, but I'm looking to use it for running some internal automation where I work.

Since you already make pre-built images for the project, would it be possible to have barebones provided on its own? It looks like that will be all that I need to make me some toast.

@piranna
Copy link
Member

piranna commented Jul 11, 2016

Well, it's a bit early to say if this will be put into production, but
I'm looking to use it for running some internal automation where I work.

I would be really happy to hear that ;-)

Since you already make pre-built images for the project, would it be
possible to have barebones provided on its own? It looks like that will be
all that I need to make me some toast.

Yes, each layer is independent, so you can be able to use nodeos-barebones
standalone, although I recomend you to do your changes on nodeos-initramfa.

@piranna
Copy link
Member

piranna commented Jul 15, 2016

Hi @sbuller, did you get any chance to take a look on this, or was nodeos-barebones useful to you? If you have any question don't doubt to ask.

@sbuller
Copy link
Author

sbuller commented Jul 15, 2016

I'm still poking around. I feel like I'm getting close to having something that does what I need.

@piranna
Copy link
Member

piranna commented Jul 15, 2016

I'm still poking around. I feel like I'm getting close to having something that does what I need.

I have described a possible solution at NodeOS/nodeos-initramfs#4, tell me if we have some points in common here.

@piranna piranna mentioned this issue Jul 16, 2016
@piranna
Copy link
Member

piranna commented Aug 5, 2016

@sbuller, did you get any progress on this?

@sbuller
Copy link
Author

sbuller commented Aug 8, 2016

I've gotten a bit sidetracked. I put together nos-init to generate initramfs files in the manner I envision. I've played a bunch with syslinux and other bootloaders, as well as related filesystems. I can't say that it's truly worthwhile, but I had been focused on generating a bootable disk image without depending on tools external to NodeJS/NPM. I settled on syslinux with fat16 is the most realistic option. It took me a while to come to the realization that there was no practical way to produce the output entirely with streams. I've got a bunch of spaghetti code at this point that puts together a lot of this but it's super hairy, and it's still missing some important parts. The stupid thing is that it's probably just a couple lines of bash script using mkisofs. I'd just rather make program that could run everywhere. That's one of the main things I like about NodeJS.

Anyhow, it's still pinging around my brain at the moment, but hasn't seen any work for a week or two.

@piranna
Copy link
Member

piranna commented Aug 8, 2016

I've gotten a bit sidetracked. I put together nos-init to generate initramfs files in the manner I envision.

That's awesome! :-D A generator of nodeos-initramfs custom images, Very neat... :-) We could spand it later to make modules run inside an unpriviledged environment, but that's definitely a good starting point, good job! ;-)

I've played a bunch with syslinux and other bootloaders, as well as related filesystems. I can't say that it's truly worthwhile, but I had been focused on generating a bootable disk image without depending on tools external to NodeJS/NPM.

That would be cool if it helps to make NodeOS build environment more portable, although by far I have been seeing that the most portable language at least for low-level and platform independent components is bash. Look, it now has official suport on Windows 10, too... :-P

I settled on syslinux with fat16 is the most realistic option.

Is that the reason why you are doing the require() magic on your code? To evade problems with symlinks on FAT16?

A bootable NodeOS image on a FAT pen drive booting directly to a secure environment, that's directly the USB drive root directory, that would be interesting... OverlayFS would have problems with that, but maybe we could be able to override them with FUSE or an upgraded version of UMSDOS... Hum... :-)

@sbuller
Copy link
Author

sbuller commented Aug 8, 2016

My intent is only to have the initramfs and barebones on the fat16 image. My use case is only for qemu. That said, I do try to keep things modular and general where I can. I can't remember my reasons for abandoning symlinking, but it's not because of fat16. The initramfs is still a cpio which populates a tmpfs.

@piranna
Copy link
Member

piranna commented Aug 8, 2016

My intent is only to have the initramfs and barebones on the fat16 image.

Seems you are creating a /boot partition, that's just the purposse of
nodeos-rootfs, but will be removed after 1.0 and integrated in the general
build process of NodeOS.

My use case is only for qemu.

In that case, QEmu can directly load and run the linux kernel and the
initram as parameters so there's no need to create a FAT16 partition image
to host them, in case of doubt...

That said, I do try to keep things modular and general where I can. I
can't remember my reasons for abandoning symlinking, but it's not because
of fat16. The initramfs is still a cpio which populates a tmpfs.

Yes, I read the code, I like your approach :-)

@sbuller
Copy link
Author

sbuller commented Aug 9, 2016

I actually really like the idea of using qemu to directly load the kernel and initrd, but using a disk image fits better with my current setup. Using disk images I can run my NodeOS images in the same way as I run all my other images.

@piranna
Copy link
Member

piranna commented Aug 9, 2016

Ok, that totally makes sense :-) How are you creating the NodeOS images?
Maybe we can improve our build process too...

El 9/8/2016 17:18, "sbuller" notifications@github.com escribió:

I actually really like the idea of using qemu to directly load the kernel
and initrd, but using a disk image fits better with my current setup. Using
disk images I can run my NodeOS images in the same way as I run all my
other images.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#273 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAgfvuZHZqKHtNf9P-hIxsfvPRwTWy7iks5qeJongaJpZM4JIbq3
.

@sbuller
Copy link
Author

sbuller commented Aug 9, 2016

Well the overall idea is to create a fat16 image containing barebones and the initramfs files, and make it bootable using syslinux. That's basically it. To do it without depending on external programs like mkfs.vfat and syslinux (or mkisofs which would be an alternative) involves downloading a syslinux file and extracting ldlinux.{bss,sys,c32}. These are then combined into a fat16 image. There's still a couple tricks I've got to figure out with that last part, but I've worked all the basics out.

A fat16 image has 4 main parts:

  1. The bootsector - ldlinux.bss forms the bulk of this.
  2. The fat - really simple, but annoying structure. Requires information about where the files are.
  3. The root directory. Requires information about where the files are.
  4. File data. Where to align these is the only question.

The annoying thing about this is it basically needs to be written back to front, which also makes it difficult to use space efficiently. I tried https://github.com/owenson/tiny-linux-bootloader, but couldn't get it booting at all.

@piranna
Copy link
Member

piranna commented Aug 9, 2016

These are then combined into a fat16 image

How are you creating the FAT16 image? Are you using the genfatfs module... or have you created a FAT16 formater in Javascript?!? O_O

The annoying thing about this is it basically needs to be written back to front, which also makes it difficult to use space efficiently

Maybe you could not be able to use the Streams API, but you could be able to create first the data section and later add to the front the filesystem structures...

I tried https://github.com/owenson/tiny-linux-bootloader, but couldn't get it booting at all.

Really interesting, specially the part of having the kernel and the initrd separated from the partition, this would be really useful to create single-user bootable pendrives hidding the kernel and the initrd files from the user when used as a regular USB drive :-) How far did you get with that? Could be the problem that it uses a initrd file instead of a initramfs?

By the way, have you written more extensively how you have done this on a blog or something? I would be interested on read it on detail :-)

@sbuller
Copy link
Author

sbuller commented Aug 9, 2016

or have you created a FAT16 formater in Javascript?!? O_O

Just about ...
The approach I'm taking makes several assumptions to keep it simple:

  1. no subdirectories
  2. filesystem size will not exceed 512 * 64 * 65525 [sic] bytes + fs data.
  3. no long filenames
  4. no special attributes
  5. no dates and times
  6. all files to be added will come from existing file descriptors
  7. ... more?

Some of these aren't hard to fix, but I'm not planning on working on them.

@piranna
Copy link
Member

piranna commented Aug 9, 2016

Just about ...

So yes, you are doing a (limited) FAT16 formatter in Javascript... :-D It's a shame can't be done streaming but makes sense, usually filesystems structures are at beginning. I don't know if the root filesystem could be put at the end...

Some of these aren't hard to fix, but I'm not planning on working on them.

I find the limitations acceptable for that use case. Are you leaving zeroes in the fields that you are not initializing so others can fill the holes later? It would be a good alternative to the usage of genfatfs, and also runtime.js guys would be interested on it... :-)

@sbuller
Copy link
Author

sbuller commented Aug 9, 2016

I'm about to pack it in for the day, but you can have a look at what I'm doing:

https://github.com/sbuller/nos-mkfat

@piranna
Copy link
Member

piranna commented Aug 9, 2016

I have been taking it a look, it's neat :-D

@sbuller
Copy link
Author

sbuller commented Aug 10, 2016

Well, it's looking like the filesystem writing is working. It looks like a couple variables need to be changed in the bootsector code when it's written though.

@piranna
Copy link
Member

piranna commented Aug 10, 2016

Couldn't them be calculated in advance?

El 10/8/2016 21:13, "sbuller" notifications@github.com escribió:

Well, it's looking like the filesystem writing is working. It looks like a
couple variables need to be changed in the bootsector code when it's
written though.


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#273 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAgfvmqv3X8YK_9X2sy0RCroYfWO7LPQks5qeiLzgaJpZM4JIbq3
.

@sbuller
Copy link
Author

sbuller commented Aug 10, 2016

I'm not sure. I'm looking into 3x 4-byte discrepancies in the boot code at offsets 0x11A, 0x120 and 0x12F. 0xDEADBEEF and 0xFEEDFACE definitely look like placeholders. I may be able to simply hardcode some values, but I'm not sure what I'm looking at. I'd managed to boot an image I put together by hand last week, so I have some hope.

@piranna
Copy link
Member

piranna commented Aug 11, 2016

That offsets are inside the actual boot code, so I doubt they are other thing that instructions in machine language. That values can be both placeholders or guard marks, so if it works without that changes, I'll left as is. What are the actual values you see different both before and after?

@sbuller
Copy link
Author

sbuller commented Aug 11, 2016

It turns out ldlinux.sys also needs some processing. I'm starting to second guess this approach. I must have been playing with already processed files and didn't manage to deviate far enough from the parameters. Maybe. I don't know.

I've been reading through the syslinux source; for reference the relevant code seems to be the function syslinux_patch() in the file libinstaller/syslxmod.c. Trouble is, even if I work out all the modifications I need to make, what happens with the next release of syslinux? And here I thought generating the filesystem from scratch would be the hard part.

This will probably have to stew in my head for a while. I'll keep reading syslinux source for now, but I don't see a way to get syslinux working quickly. At least the fat16 code is theoretically useful even as it is.

@piranna
Copy link
Member

piranna commented Aug 11, 2016

I'm not sure what problem do you have with SysLinux, but could it be that you are generating a partition image instead of a disk image and SysLinux can't be able to manage that in a cleanly way?

@sbuller
Copy link
Author

sbuller commented Aug 11, 2016

No, if I run the syslinux program on the fs/disk image that I generate it works fine. The issue is that I've been trying to put the files syslinux uses on the fs myself, rather than letting syslinux do it. It turns out syslinux doesn't just copy a couple files and write a generic boot sector—it does those things, but then makes a few changes. Anyhow, it looks at this point like the tiny-linux-bootloader is a much more attractive option.

@piranna
Copy link
Member

piranna commented Aug 11, 2016

The issue is that I've been trying to put the files syslinux uses on the fs myself, rather than letting syslinux do it. It turns out syslinux doesn't just copy a couple files and write a generic boot sector—it does those things, but then makes a few changes

I use the installer, so can't help you :-( Maybe you would exec it in a second step, or ask SysLinux guys about what changes are they doing...

Anyhow, it looks at this point like the tiny-linux-bootloader is a much more attractive option.

Lol! X-D At least it's an interesting one... :-)

@zimme
Copy link

zimme commented Oct 7, 2016

This is awesome, keep up the good work guys =)

I've had this idea that I wanna build NodeOS docker images that just runs my apps, this way I can just have my CI server run a build step on git pushes and deploy the generated docker images to a private docker image registry and then I have a registry of versions of my app as docker images available to deploy from =)

@piranna
Copy link
Member

piranna commented Oct 9, 2016

I've had this idea that I wanna build NodeOS docker images that just runs my apps, this way I can just have my CI server run a build step on git pushes and deploy the generated docker images to a private docker image registry and then I have a registry of versions of my app as docker images available to deploy from =)

Well, you are in good news, because due to popular demand we have been adapting NodeOS to make it possible to use as a single-process OS, and now it's possible to build custom initramfs images running the modules in an isolated environment. It needs some testing though, but probably is what you are looking for your Docker CI system.

Your system seems to be very interesting, if you need some help to adapt NodeOS to it or to integrate it in your system I can offer you some contractor advice, so if you are interested don't doubt on send me an email :-)

@piranna
Copy link
Member

piranna commented Jan 18, 2017

With the new structure on RC3 it would be trivial to make nodeos-initramfs module to build a system like this, do you want to take a look? :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

4 participants