btrfs root filesystem

[CurlyMoo]

I will send you image of xbian.alpha5, converted to btrfs, with @distro plain snapshot, separate /home with same snapshot, standard initramfs (holding splash as original), with part resize upon start on different media update to mount all and upstart, removing sysv layer and service which is able to revert to any state of / or /home separately. /home without reboot, / with just reboot with nodelay.

only issue was with xbian-update package holding some libs for cec, which apt removed (dependency to sysv layer because of boot local script).

libs are back including splash and images, but only copied.

• Creating a new update that doesn't depend on console-tools
• Compile new kernel
• Creating a new filesystem structure on btrfs with / and /home, and a /boot fat partition and copy XBian onto the new structure
• Implement the new ram / swap logic
• Reimplement the initramfs with the resize logic adapter to the new structure.
• Implement upstart (or systemd?)
• Implement the new snapshot / backup functionality
• Describe all steps on our wiki, so new developers know exactly what has been done.
• Thinking about live converting of partitions that i believe is possible with btrfs (AWESOME)*

[CurlyMoo]

Summary

You created 3 partitions, which all contain a btfs filesystem?

1. /boot
2. /root
3. /home

The resize on boot script resized only the /home partition and leaves the /boot and / partition as they are.

The restore of a snapshot of home without a reboot is really cool, just like i know it from ZFS.
To restore a snapshot of the root partition we can add a system restore point just like in windows. Then we can add an additional cmdline.txt parameter like systemrestore=GMT-2013.03.01-14.00 which restores to that specific snapshot.

only issue was with xbian-update package holding some libs for cec, which apt removed (dependency to sysv layer because of boot local script).

Did you implement upstart and thereby remove initd, on which klogd and syslogd are dependent.

[mk01]

I had already three times all written, how is done. But was so much text, I deleted and decided to send the image.

Concept is the same as a5 release. two parts /boot and btrfs volume the rest. there, two subvolumes are created, one is /, second one is /home. for both, two snapshots exists. one called @distro (represents initial clean image) and @running. upon boot, @running is mounted, for home the same applies.

Resize runs as upstart process, will resize partition#2 and btrfs volume (it's started in paralel to other processes, not during initramfs phase). is independent.

To reset to factory, you can reset xbmc (means reset only subvolume /home). you just stop xbmc, umount /home, delete snapshot @running, create new @running from @distro, mount /home and ready. takes 10seconds.

To restore system state (roots), you do the same, but until you reboot, the change is not effective. but steps are the same. And reset hard is do it for rootfs and home as well.

to be honest, I was going through old issues, and all the scripting done by you and team just to fix absurd issues. I was figting with sysv start design since redhat 5 back in last century. On top I wanted to speedup starting and runlevel concept with strict ordering and serialization is not able to provide this. So I decided to convert to upstart and mountall package (which I know very well, I was helping with porting to zfs support). So i chosen the faster and for me fine solution (anyhow sooner or later this will happen). RH like distros introduced systemd, debian upstart, both are still taken from apple's darwin 10years old system.

but works better as soon as you starts init->udev->mountall no more waiting on errors stopping whole boot etc, not connected lans, unable to mount / start something.

Anyhow I installed upstart bridge to init.d scripts, so they are started with no issues as well (but according to upstart policies, not sysv). requirements are still valid you can simply take what upstart script emits and put into prerequisite. no problem. So if this bootlocal script would not be inlcuded in the package, nobody be realised the change.

btw: klog syslog and others are still as init.d scripts starting with no issues, still possible to work with them as before. But are started in upstart enviro. You don't see the difference (beside better startup process and better control over conditions for start / timing / fail)/. But from dpkg point of view is sysvinit, initsctips are conflicting to upstart and mountall. So what I didn't rewritten to upstart, is still there started by this "bridge upstart service", but for dpkg this functionality is no more there. I found until now two packages, which failed to install automatically - although only issue was the startup scipt. But if you cp to /etc/init.d, works as usual. Maybe I will create some king of meta package to "provide" what's there, but dpkg thinks opposite.

Ah, one more thing, swapping to CoW filesystems is technically problem. When I started, I created parition. But then remembered zram, so currently now swapping goes into compressed memory. Due to this I also altered few kernel parameters, swappines, dirty region writes, and pressure from buffers / caches.

To speedup the mmc/usb operations, i'm delaying the writes growing fs cache, if there is ram, but changed the pressure from apps pages buffer to push to fs cache and reclaim faster when needed. I also used tmpfs for run lock and shm. Runs quite nice, my fully configured xbmc starts now within 25s on PI (inlcuding xbmc loaded in home screen).

btw: how is the splash working? you have to kill it always and start new? When I turned the boot splash from initramfs, new splash is started with xbmc. And both were running all the time with xbmc in paralle l, taking 15-20% of cpu. So I added a script to XBMC to kill splash processes when xbmc starts and takes the screen. Maybe I introduced this somehow.

To upload the image, do you have some "public" space where I could upload ?

[mk01]

I installed cpu freq scaler as well, with extreme over clocking makes diff with top temp. runs 5C cooler by returning to base 700mhz.

[CurlyMoo]

I had already three times all written, how is done. But was so much text, I deleted and decided to send the image.

Thanks for taking the time to write it again!

Concept is the same as a5 release. two parts /boot and btrfs volume the rest. there, two subvolumes are created, one is /root, second one is /home. for both, two snapshots exists. one called @distro (represents initial clean image) and @running. upon boot, @running is mounted, for home the same applies.

Just like ZFS (i don't know btrfs)

Resize runs as upstart process, will resize partition#2 and btrfs volume (it's started in paralel to other processes, not during initramfs phase). is independent.

Why not running the resizer inside the initramfs?

To reset to factory, you can reset xbmc (means reset only subvolume /home). you just stop xbmc, umount /home, delete snapshot @running, create new @running from @distro, mount /home and ready. takes 10seconds.

To restore system state (roots), you do the same, but until you reboot, the change is not effective. but steps are the same. And reset hard is do it for rootfs and home as well.

Sounds really great!

to be honest, I was going through old issues, and all the scripting done by you and team just to fix absurd issues.

Can you give examples. I already started fixed some wierd issues from the pre 1.0 era but i'm not the full blown linux professionals as some xbian members think :)

I was figting with sysv start design since redhat 5 back in last century. On top I wanted to speedup starting and runlevel concept with strict ordering and serialization is not able to provide this. So I decided to convert to upstart and mountall package (which I know very well, I was helping with porting to zfs support). So i chosen the faster and for me fine solution (anyhow sooner or later this will happen). RH like distros introduced systemd, debian upstart, both are still taken from apple's darwin 10years old system.

I was planning to install systemd because i thought it was easier than upstart, however never worked with another init then initd. If you know upstart and can implement it for us, we would be really cool!

but works better as soon as you starts init->udev->mountall no more waiting on errors stopping whole boot etc, not connected lans, unable to mount / start something.

I don't understand this part...

Anyhow I installed upstart bridge to init.d scripts, so they are started with no issues as well (but according to upstart policies, not sysv). requirements are still valid you can simply take what upstart script emits and put into prerequisite. no problem. So if this bootlocal script would not be inlcuded in the package, nobody be realised the change.

What where the issues you encountered then with our current packages?

btw: klog syslog and others are still as init.d scripts starting with no issues, still possible to work with them as before. But are started in upstart enviro. You don't see the difference (beside better startup process and better control over conditions for start / timing / fail)/. But from dpkg point of view is sysvinit, initsctips are conflicting to upstart and mountall. So what I didn't rewritten to upstart, is still there started by this "bridge upstart service", but for dpkg this functionality is no more there. I found until now two packages, which failed to install automatically - although only issue was the startup scipt. But if you cp to /etc/init.d, works as usual. Maybe I will create some king of meta package to "provide" what's there, but dpkg thinks opposite.

Ah, one more thing, swapping to CoW filesystems is technically problem. When I started, I created parition. But then remembered zram, so currently now swapping goes into compressed memory. Due to this I also altered few kernel parameters, swappines, dirty region writes, and pressure from buffers / caches.

To speedup the mmc/usb operations, i'm delaying the writes growing fs cache, if there is ram, but changed the pressure from apps pages buffer to push to fs cache and reclaim faster when needed. I also used tmpfs for run lock and shm. Runs quite nice, my fully configured xbmc starts now within 25s on PI (inlcuding xbmc loaded in home screen).

I'm just really direct here. Would you like to take over some of my intermediate level of XBian scripting to make it more professional? I do not fully understand everything you write but do believe you have some massively great knowledge that we can use. I can help you with implementing them by given you the permissions you need on our git repositories. I can also explain all choices we made and how the system currently works. Just let me know if you would like to do such a thing.

btw: how is the splash working? you have to kill it always and start new? When I turned the boot splash from initramfs, new splash is started with xbmc. And both were running all the time with xbmc in paralle l, taking 15-20% of cpu. So I added a script to XBMC to kill splash processes when xbmc starts and takes the screen. Maybe I introduced this somehow.

The splash was my proof of concept but far from perfect. The splash is located on the normal rootfs. When the initramfs has mounted the rootfs partition it starts the splash and loads the images that are needed into the ram. It will get some status process updates from some init scripts that where places in between the boot process. It should not start multiple instances but as i told you, the splash is far from perfect but working. I would really like to have a ability to know the actual boot process so the splash can be a lot more verbose and precise. The source can be found here: https://github.com/CurlyMoo/Splash. We also need some real C(++) professionals to rewrite it to a fully functional one.

To upload the image, do you have some "public" space where I could upload ?
Can you upload it to dropbox or mediafire or such?

I installed cpu freq scaler as well, with extreme over clocking makes diff with top temp. runs 5C cooler by returning to base 700mhz.

The CPU scaler was there by default however i made the performance governor the default in the kernel. My experience was that dynamic switching of the CPU frequency was to slow to have a responsive system. By defaulting the performance governor gives that snappy touch to XBian and the temparature isn't really an issue for the SoC.

[CurlyMoo]

Also, i can ask if we can give you access to our developer forum, but it depends on my question if you would like to do some developing for the (linux) core of XBian

[mk01]

sorry, i'm not native speaker and if I rush the text, I can't understand after myself as well.

I like systemd more as well, because personally I'm runnig Macs and there is existing called launch-daemon more than5 years and few generations of OS. So it to me familiar the concept, thus the linux implementation is so much chaotik, I was always looking hours to find a specific setting.

upstart looks at first easy like "init.d" ver2., directory is /etc/init, everything is there, but scripts are not scripts (runnable), just config files started with preloaded - which is consolidating signals / prerequisites / dependencies and configured needs and starting in parallel with no blocking. You will see.

I have my PI few days, and only system I started with was xbian.a5 and have seen dmesg with the governor compiled in kernel. And was really thinking, why is nobody using for PI :). The image I will give you is a usb key I'm booting directly and via USB is not snappy 100% of time. The CopyOnWrite design of btrfs needs bandwidth and on my PI the usb key is doing not more than 5MB/s. MMC 25MB/s.

With the "reset" feature I will continue to develop on the same MMC card, I will just change the /home for other. Is faster than reboot to different system.

What I have not tested is the SQLite databases. I have 6 XBMC installations and shared mysql db. So I'm not sure how it runs on btrfs. Theoretically, if db has 100mb, if you change just one bite of info, CoW means that complete data will be copied first, then the previos copy will be deleted. It's possible to disable it as mount option, but I have no test data.

btw: I have seen the service scripts set to start at 25ft, 50th, etc. I was really thinking few hours, how you do the percentage during initfs phase. ThumbUp for this simple idea, really. Unfortunately, the sysvinit process is the only one, where it's going to work :)) because of it's strict ordering and serialization.

And yes, I'm always pleased if there is something to do, like this.

And last but not least, there was a package in the image xbian.alpha5, called xbian-update. although it's direct dependencie is not sysvinit package, look, what happened.

'''
Start-Date: 2013-03-16 05:25:36
Commandline: apt-get install upstart mountall cpufrequtils plymouth
Install: libnih-dbus1:armhf (1.0.3-4.1, automatic), libcpufreq0:armhf (008-1, automatic), libjson0:armhf (0.10-1.2, automatic), makedev:armhf (2.3.1-92, automatic), upstart:armhf (1.6.1-1), mountall:armhf (2.46), cpufrequtils:armhf (008-1
Upgrade: udev:armhf (175-7, 175-7.1), libudev0:armhf (175-7, 175-7.1)
Remove: console-tools:armhf (0.2.3dbs-70), xbian-update:armhf (1.0-0.5), sysvinit:armhf (2.88dsf-34)
Error: Sub-process /usr/bin/dpkg returned an error code (1)
End-Date: 2013-03-16 05:26:01
'''

I see. It's not the upstart/sysv story. it's console-tools. and xbian-update dependency to this package caused uninstall of xbian-update what caused uninstall of initramfs, splash, libcec, sec, so hdmi control down...
So I manually extracted the files and refreshed libraries and all ok. So no real dependency on console-tools. Can you check it?

[CurlyMoo]

A little history to clarify things a bit and the progress that has already been made.

My history goes back to the 0.6 era. To understand a bit better from where we came, a little history. The major problems with the previous XBian < v1.0 versions where:

• Init scripts weren't used but everything boot logic was put in the /etc/profile file.
• Everything ran as root and due to filesystem permission issues sudo didn't work and wasn't re-installable.
• Updates where done with descructive tarballs.

I just made a framework (with a lot of proof of concepts) so now all kinds of developers are now volunteering to improve small parts. The most important things i did where:

1. Making XBian real open source by posting it on git allowing easier development by new developers.
2. Rewriting XBian again on top of a Raspbian image to fix all those wierd implementations of < v1.0 such as:
   * Filesystem permissions where back as they how they supposed to be
   * I implemented and wrote init scripts for our new programs/logic
3. I wrote xbian-config for easy configuration --> this was later pick-up by another developer who created an xbian-config inside XBMC with the basis the xbian-config-shell.
4. I made the basis for the initramfs and splash (first animated splash on the raspberry pi) which is also attracting new ideas and developers.
5. Implementing a new kernel
6. Created an apt-repository for package handling which will be automated in a short while so external (trusted) contributers can directly add packages.

As you can see, i already made a lot of changes in professionalizing XBian, which i could do with the time i had back then (for about 3 months). My time has been scarce again, so instead of a couple of days of developing a week, i went back to a only few to no hours a week. I also busy with supporting and seeking new developers/contributers to take over some parts or improving things, so we all do what we do best and have to do less so the project doesn't consume all our time.

And example of the framework i talked about earlier is the download package. I created the combination of packages by patching and configuring them. However, that was about it. I was not going to add more things myself so we advertised for new package maintainers. A new contributer has taken that role and has added new functionality and already released new versions of packages through our apt repository. So that's how thing go at the moment.

sorry, i'm not native speaker and if I rush the text, I can't understand after myself as well.

What is your native language?

I see. It's not the upstart/sysv story. it's console-tools. and xbian-update dependency to this package caused uninstall of xbian-update what caused uninstall of initramfs, splash, libcec, sec, so hdmi control down...
So I manually extracted the files and refreshed libraries and all ok. So no real dependency on console-tools. Can you check it?

I can create a new update (just for this purpose) that doesn't depend on console-tools. This way, you don't lose the current packages when implementing the new features.

And yes, I'm always pleased if there is something to do, like this.

Let us think about a best way on how we can work together in implementing all your great ideas... My suggestion would be to do these things in small steps. So, what about this:
Check the FP

I will start creating the new xbian-update package and will share that asap.

Can you share what your changes to the current kernel where so i can implement them? Or do you want to take over the kernel compilation (which is fine by me)? I have a cross-compiler that you can use in (a virtualized) Ubuntu. On a 2500K it takes about 15m to do a full kernel compilation. I can also provide you with all deb creation scripts.

*This way we can keep providing updates through our apt-repo instead of forcing users to reinstall.

[CurlyMoo]

About the filesystems, are those as how it would be done with ZFS. We create a single RAID 0 btrfs pool with the SD card as the single device. Then on top of that pool we create two new filesystems as in / and /home. Is that correct?

[mk01]

I was hoping for the same and much confused, when trying the "sub volumes" and "snapshots" zfs has really one pool (pure capacity) and separate filesystems - hierarchical structure (and lack of a hierarchy is really making btrfs unusable in large scales). btrfs is just one volume (mount settings and structural settins are shared for all subvolumes. so it's more like common space with possibility to mount separate substructures as separate fss. So like btree within a btree. Nothing more.

under ZFS you have /. then /home. If you promote snapshot of / to root and this root has sibling /home, you are sure, that rootfs will change (you changed snapshot), but /home will be where it should be! under /home. but btrfs, you have root. you make a subvolume /home. After, you create snapshot of /. And if you remount to this new snapshot, you can see /home. Because was sibling of another hierarchy.

The hierarchichal bond is hard within the btree structure. Snapshot of root is snapshot of the data, but other connections are gone. with old structure.

ZFS has hierarchy, you see it as a tree. But the connection is mountpoint. Static value. Yes, you can change it and so move where you wish, but if once at /home, it is under home. Regardless of version / snapshot / or anything else.

But still good for our needs and dynamic structures.

[mk01]

I'm uploadig the image to uschovna.cz, link will be emailed to info@xbian.org

[mk01]

http://uschovna.zoznam.sk/download?code=1342685841985-TXwjUKwAtDzxrLfBpWcA

[CurlyMoo]

Here is the new update:
http://jb.xenserver.sk/xbian/xbian-update1.0-0.5.deb

[mk01]

Curly, can you pass xbians std kernel config? Anyhow finally solved the issue with dependancies. Created dummy package (sysv scripts) for old sysv package and restored all the packages with apt-get with no issues.

[CurlyMoo]

Curly, can you pass xbians std kernel config?

The most important information about kernel compilation can be found here:
https://github.com/xbianonpi/xbian/wiki/Compile-Kernel

And here:
https://github.com/xbianonpi/xbian-patches/tree/master/kernel

However, the code to include authfs wasn't documented yet, but if we are going to use btrfs, this will become obsolete i think.

Created dummy package (sysv scripts) for old sysv package and restored all the packages with apt-get with no issues.

So, what changes did you make to our clean Alpha 5 image?

[CurlyMoo]

I've uploaded the xbian kernel source on our git: https://github.com/xbianonpi/linux so you can check what changes / patches you would like to see implemented.

[Koenkk]

@mk01 the beta 1 update is here: http://dikonist.asuscomm.com:82/XBian/xbian-update1.0-1.deb , but currently breaks your system because I only applied the changes that are in xbianonpi/xbian/master