New branches 'legacy' and 'rationalise'

[01micko]

For more information, I will refer readers to this blog post.

I have created 2 new branches legacy and rationalisation.

Important

As @mrfricks and myself are in development please keep changes to testing as bugfix only or minor updates unless there is a pressing need for change. After release of xenialpup and slacko-63x (bugfix) it is intended that rationalise branch will be merged with testing and deleted. Hopefully this will only be a matter of weeks. Of course all xenialpup changes before and at release should land in testing so that we can merge that with master.

'legacy'

legacy branch is purely in place to preserve old code. Read the README before making changes to that branch - changes are discouraged.

'rationalise'

rationalise branch is for sweeping new changes including but not limited to:

• deletion of un-maintained distros (arch, mageia etc) and older unsupported versions of distros (ubuntu - jaunty, karmic, lucid; debian - squeeze; slackware - 13.1)
• rationalisation of code in the woof build system
• changes in the way fdrv is dealt with. See fdrv appears in DISTRO_SPECS but is not loaded #785
• corresponding changes to kernel-kit and rationalisation of code
• initrd_progs
• other stuff which may need enhancing. Suggestions welcome.

Notes

I expect that @zigbert will keep updating his progs in testing. Therefore, before anyone makes a commit or PR to rationalise be sure that you have merged the changes in testing. This will make for minimal conflicts when rationalise is merged back into testing.

Of course any minor bugfixes and enhancements should land in testing first. So if you have a change to make, carefully consider if it is major or minor before committing.

As usual, have fun!

[01micko]

I have removed some (not referenced above are arch and mageia and slackware/13.1).

I left T2and pet-based for now as they could be used as templates for a possible sabotage branch.

[wdlkmpx]

gyro posted 2 patches in the forum

1. To add help msg for his additions to init (bootloader menu)
2. To make fdrv behave like the other *drv (not tested), but I'll test it myself.

Regarding kernel-kit, I have to offer this as a slightly improved version
http://murga-linux.com/puppy/viewtopic.php?mode=attach&id=97710

Check it out and provide your impressions about it.

initrd_progs: how exactly it should be merged? There are replacement for commands and bb-create-symlinks

I'll copy rootfs-skeleton to my system to make sure everything is ok, i'll also start building a pup to make sure I don't break anything.

[01micko]

1. @gyrog 's enhancements should be added - be aware of the newish support for UEFI too regarding the help messages.
2. I'll test your adjustments to kernel kit soon, but probably not until next weekend.
3. I think initrd_progs (for now) should be put straight in like kernel-kit - of course minus the git history. And, for now, it should be (or should that be 'maybe') optional to use it or go with the old method of using the pre-built stuff. Of course there are some hurdles - DISTRO_SPECS should be obtained from the target, plus other stuff that I can't recall right now. The timing will be critical too - in the context of 3builddistro-Z. Cross builds may be an issue. Not so much x86/x86_64 but more for x86*/arm*. It's a learning curve for us all!

[gyrog]

@01micko,
When you say "@gyrog 's enhancements should be added", which branch do you mean?
My current plan is to pull-request all my patches against the "rationalise" branch.

I actually have 5 in the pipeline.

1. Update "help2.msg", I'm not happy with the current version in the forum, it needs a bit more thought.
   The current "help2.msg" in "testing" is not wrong, it just doesn't reflect the extra flexability available now.
2. Change the way "fdrv" works, ready to go, but I think only peabee wants this now.
3. Update "help1.msg" to remove "fw_drv".
4. Tidy "init" by moving the fdrv load code to be with the zdrv load code, definitely minor.
5. Tidy "init" remove some references to tmpfs mountpoints that are no longer used.

[01micko]

My current plan is to pull-request all my patches against the "rationalisation" branch.

Indeed that is the one @gyrog 😄

2. Change the way "fdrv" works, ready to go, but I think only peabee wants this now.

With some fairly trivial changes to kernel-kit this can be facilitated. I think it is the way to go going forward. I recall you questioning 'why is firmware part of the kernel?' and this addresses that. Also helps 'libre' puppy by keeping firmware out of the kernel sfs (zdrv).

@wdlkmpx has already began adding commits, most notably restoring 'init' and 'rc.sysinit' to the original place and deleting the old 3builddistro.

I look forward to your contributions and if anything is unclear, while I am a bit busy, I'll be as attentive as practical.

Cheers!

[wdlkmpx]

I think the static binaries (initrd_progs) must be in a repo. All of them in a tar.xz archive in this format:

initrd_progs-binaries-$date.tar.xz

I want initrd_progs to be used as a standalone app as well as part of a greater scheme (3builddistro).

Perhaps a new switch: -prebuilt (build.sh will and download use already compiled binaries) and 3builddistro can call initrd_progs/build.sh this way (or another):

(
    cd /path/to/initrd_progs
    rm -f initrd.*
    ./build.sh xz -auto -prebuilt -arch arm -specs /PATH/TO/DISTRO_SPECS &>/dev/null
    mv -f initrd.gz /pathto/maindir
)

if [ ! -f initrd.gz ] ; then
    error!!
fi

[wdlkmpx]

The init script needs a refactor to allow proper translations. The current implementation runs a modified init to do so.. crazy.

Network Wizard has the right approach, it sources the the main english *.po file (actually a script with variables only, each variable contains a message), then it sources other files according to your locale. The same logic is applied in my modified MSCW.. that's the way to go.

This is something that needs to be discussed with many people, specially translators.

And those translations can go directly in the initrd-tree.. the initrd_progs repo can be used to advance this..

[gyrog]

The init script could do with more than just a refactor to facilitate translations. It could use a major rewrite to simplify it, as well.
I have some ideas on that, but maybe here is not the place to discuss it.

[wdlkmpx]

gyro I think you should open a thread about the init rewrite and the work should be done in initrd_progs... when initrd_progs lands on woof-CE some directories will be deleted and in fact that would mean a new version of woof

I strongly recommend you to "build" a initrd.gz file with a tahrpup or slacko, use it to test, and inspect the new apps and possibilities. Just run ./build.sh (will download about 150-200MB of stuff) and hit enter at any prompt (2), and then a initrd.gz file is ready.

I'm also simplifying build.sh while adding some new functionality to make it completely independent ( puppylinux-woof-CE/initrd_progs@a627c92 ), so if you have any issues, just let me know, and do add your changes as soon as you see it's working, if something is broken i'll perhaps notice it and i'll notify you.

[gyrog]

@wdlkmpx,
Thanks for the suggestion, unfortunately I haven't girded my loins sufficiently to undertake such a large project as rewriting init, yet.
Although I'm open to the possibility of discussing some ideas, so I will consider opening a thread.

[wdlkmpx]

The whole "woof-arch" should disappear or at least most if it. I can add a "pkg" to initrd_progs to compile and archive all the support/boot apps and that can be downloaded afterwards.

The commit history will grow by 15mb for no apparent reason and, i would suggest to restart in a few months but before that put the repo including commit history in a tar.xz archive ready to download for everyone to see what happened before and look for the legacy branch, though nothing of value would be lost, except perhaps Arch compat, but again and i can make it come back, but it has gone the wrong way..

I wonder what .bac files need translation into a more reasonable language, i don't understand baCon though, looks primitive like a Windows shell script

[01micko]

BaCon is structured Basic. It's fairly easy to grasp. It converts the code into C then compiles it, but the C sources produced are rather cryptic. Probably just as easy to go back to shell.

Another thing to consider is updating encryption to support dm-crypt instead of the old DES and XOR, then we can update losetup >= util-linux-2.22 instead of sticking with the ancient version which dropped support -

Util-linux 2.22 Release Notes [Sep 4, 2012]
===========================================

 The cryptoloop support in the commands mount(8) and losetup(8) is DEPRECATED.
 This is the last release where encryption= mount option and -e,-E,--encryption
 losetup options are supported.

https://www.kernel.org/pub/linux/utils/util-linux/v2.22/v2.22-ReleaseNotes

[wdlkmpx]

Yeah that should be a priority. I'm already using util-linux 2.27, the losetup in the new initrd is from util-linux 2.28... looking at the config file of the kernel i'm using, i see this:

CONFIG_DM_CRYPT=m

The agent said that this was implemented in fatdog long ago, so anybody using fatdog paste code here to look at it

As usual, the Arch wiki also provides lots of info to read https://wiki.archlinux.org/index.php/Dm-crypt

I'll look into this

[wdlkmpx]

After commit puppylinux-woof-CE/initrd_progs@f3f2944 initrd_progs is ready to be merged so I can actually face a problem that I will solve.

I will create a pull request

[wdlkmpx]

I opened a thread at the forum:

http://murga-linux.com/puppy/viewtopic.php?p=907290#907290

meanwhile micko... i was not able to get the armv6l cross compiler to work. Perhaps it works in your slacko64, type ./build.sh -arch armv6l -auto and see what happens..

[01micko]

busybox built, exited on disktype.

[wdlkmpx]

Well, slacko packages are taking forever to download. If there are 10 mirrors, on each download choose a random mirror (between 1 and 10)..

Obviously this is just me doing and thinking about everything, and it will continue that way until i shut up... so I wonder where a lengthy discussion about all of this can happen..

[wdlkmpx]

I think that by now it's safe to wipe compatibility with Red Hat or RPM based distros such as Mageia and Scientific Linux and just remove any RPM traces from the build scripts... petget should keep supporting that format, some very basic binaries should work on any distro..

It would also be a good idea to create a newer LFS Puppy, like the old one, for those who want to port Puppy by compiling themselves, but this task is huge. I know this is something already done, perhaps by fatdog? (never used it). If there is code, something already usable, we should just use it and try to improve it................

[wdlkmpx]

2createpackages can get very heavy when processing packages with lots of files, reminds me of the new2dir script, the same logic and heavy processing. It tries to strip binaries.. and that happens again in 3builddistro-Z

Half the code in 3builddistro is commented out... should be deleted.

I see repetitive code in all the build scripts, this can be simplified by sourcing a script with functions..

[dimkr]

I think we should start with mass-scale code deletion, before we do any refactoring. Something in the spirit of grep -l magia $(find . -type f).

[wdlkmpx]

Yes of course, time to delete

[dimkr]

I created pull requests for Scientific/RHEL and Arch. Low-hanging fruit.

[01micko]

If you want to merge rationalise into testing, then go for it. I would say that it's at alpha stage.

Bear in mind that @mrfricks is trying to push a reasonably stable xenial pup. I would leave testing alone, let him get that out the door then go for it.

[gyrog]

@peabee, Thanks for the iso, downloading now, I'll give it a go.

[gyrog]

What copies rc.sysinit and rc.update to the save-layer?
I just did a fresh frugal install of peabee's iso, created a savefolder, changed some configuration, several reboots.
And then I noticed that my LxPupScsve/etc/rc.d/ contained copies of rc.sysinit and rc.update that are identical to those in puppy_LxPupSc_16.08.992.sfs.
I did not attempt to edit them, so I did not do any thing to induce an aufs copy up.
It's not a rogue rc.update doing an unnecessary version update, since there was no rc.services etc...
After deleting them from a different puppy, they did not come back, after several reboots.
What's happening here?
Why does it matter?
Because it's a trap waiting to spring when someone tries to test a new rc.sysinit or rc.update by adding it in a ydrv.

[01micko]

Ok, so I'm running slacko64-6.9.5 built from slackware64-14.2 packages and it's all running pretty smooth. loaded libreoffice and @peabee 's latest chromium64 sfs's and no dramas there, even after rebooting a couple of times. I'll upload this for broader testing.

# grep -i 'build' /etc/DISTRO_SPECS 
#The distro whose binary packages were used to build this distribution:
#The version of the distro whose binary packages were used to build this distro:
BUILD_FROM_WOOF='rationalise;f045eca;2016-08-28 09:09:08 +1000'

The very latest (but I have a couple of patches to upload later today - slacko specific)

Probably take a few hours to upload on my brilliant connection.. might even use my phone so it's faster.

[01micko]

Bugger.. nearly as slow on my phone - 4G 😕

[01micko]

@gyrog I see that bug too. Also functions4puppy4 is copied and rc.country

[wdlkmpx]

I don't see where it happens. in rootfs-skeleton i see greps in sfs_load (to be deleted) and a few other scripts.. but is that enough? Probably something outside woofce or maybe an aufs bug? Does this also happen in tahrpup gyro?

[wdlkmpx]

In the rationalise branch sfs_load does not even grep rc.sysinit. actually nothing touches rc.sysinit, pmodemdiag greps a temp rc_sysinit. There is no evidence..

[01micko]

anyway, slacko64-6.9.5 alpha is released .. http://www.murga-linux.com/puppy/viewtopic.php?p=920437#920437

[gyrog]

The rc.* copyup "bug" is part of first boot in LxPupSc-16.08.992, I'll try tahrpup 6.0.5.
Edit:
The result is the same in both tahrpup 6.0.5 and LxPupSc-16.08.992, on first boot, at the end of init, booting pfix=rdsh, only BOOTCONFIG and PUPSTATE are there, "exec switch" to desktop, "X" out of the 2 firstime config screens, and now the drifters have taken residence.

[01micko]

looks like they get copied up in rc.update... if they were never copied in the first place then that code wouldn't be needed.

[01micko]

Or maybe not but the fact that that code is in rc.update shows that it is intentional.

[gyrog]

In tahrpup 6.0.5, rc.update only copies rc.sysinit if it thinks it's a version update, and then it also copies lots of other stuff including rc.services.

[01micko]

I just did a pristine boot of my latest slacko and there is a huge clue as to why these files are copied up at initial boot in /tmp/bootsysinit.log

cheching if interface eth0 is alive...Files being translated:
 /etc/rc.d/functions4puppy4
 /etc/rc.d/rc.sysinit
 /etc/rc.d/rc.update
 /etc/rc.d/rc.country
 /usr/local/petget/0setup
Files being translated:
 /root/Choices/ROX-Filer/PuppyPin
 /root/.jwmrc-tray

So I guess it is the SSS or whatever it is that translates files.

[01micko]

So.. when quicksetup is run then SET_LOCALE="yes" .. therefore fixscripts always runs whether the file needs translation or not.. so is copied up.

Ok, we have the cause but the solution may not be so easy.

[gyrog]

But locating the cause is a good start.
So if I stop quicksetup from running, there should be no copy-up.

[gyrog]

All DISTRO_IDSTRING processing in Woof-CE rationalise can be removed.
It's not used at all in the new init, so all it achieves is a couple of warning messages from losetup in /initrd/tmp/bootinit.log. Everything works fine after I "head -c -32" the offending files.

[gyrog]

I just created a ydrv conatining the flag file in /etc/ that stops quicksetup from running, then did a first boot of LxPupSc-16.08.992, I didn't get any quicksetup screen, and I didn't even close the welcome screen, simply looked in /initrd/pup_rw/etc/rc.d and the 5 squatters were there.

[gyrog]

The "bug" occurs somewhere in rc.sysint, they're not there at the beginning, but they're there at the end. The bootsysinit.log also contains messages abot translating these files just after eth0 network message.
Later Edit:
Well it happens while rc.sysinit is running.
bootsysinit.log.gz
In the attached bootsysinit.log my "ls of /initrd/pup_rw/etc/rc.d" message can be seen towards the beginning and towards the end. The lines following are the output of "ls /initrd/pup_rw/etc/rc.d".
As can be seen the files are not there in the beginning but are at the end. And there's that "translated:" reference.
The problem is that looking at rc.sysinit, there is nothing between "RECOGNISE MEDIA DEVICES" and "MISC. DESKTOP STUFF" that does that. So it looks like it's something running in the background.
The difficulty here is that the only stuff that should be running in the background is the rc.services stuff, and there's no obvious canidates in /etc/init.d
??????

[peabee]

Xenialpup-7.0.4 has the "bug" - DISTROSPECS doesn't say which version of woof-ce it's built with.....

[gyrog]

It looks like 'fixscripts' is being run. The question is by what and why?
'fixsctipts' is sometimes run in rc.update, except that code should only be run for a version update, and pupmode=5 is supposed to exit before it even reaches that code.
Maybe the real solution is to get rid of this "dynamic" translatioin of these scripts and discard the whole fixscripts stuff.

[wdlkmpx]

Many scripts need to be audited, some pivotal scripts should be in rootfs-skeleton, i see why development stopped at some, it's probably a matter of geological eras.. I see James C reported an important bug, he has many partitions and a distro installed in each one of them. So it's probably a good idea to create many partitions in a usb with different filesystems to see the performance..

[gyrog]

If you're talking about the "jumbled" drive icons, then robin2 has it right, they're not jumbled, the partition names are just sorted alpabetically. A local comparision function might be required.

[gyrog]

Re eCryptfs for save layer encryption:
It's a no go. An encrypted directory mounted as "plain text" is rejected by aufs as a branch.
But if there is an encrypted directory within the stack, it can be mounted as "plain text" by referenceing it via the stack. So it could be used to encrypt a data directory.

[wdlkmpx]

So... plan B? crypsetup?

[gyrog]

There is no plan B.
Maybe I'm just rationalising, but why encrypt the whole save layer. A significant portion of which is likely to be public software installed via .pet's or the PPM.
If I thought I had some files worth hiding, I'd go for an encrypted directory containing just those files, that only gets decrypted when and if I want to. And that's very doable with ecryptfs in puppy.

[wdlkmpx]

so it seems a "rebase" to fix something caused a little trouble. i'll add a script to download and setup a local woofce git repo in a few seconds. i'll just edit the one i have which i've used it for all the repos i contributed to before.

i think the petbuilds should be in the main repo.. or at least a "special" petbuilds.. the reason is that i want to add some "standard" builds... busybox, util-linux, coreutils, wget etc.. to be compiled one by one. this will bring consistency and would help older woofce distros.. the templates should follow the same logic regarding what apps are chosen/discarded.

[wdlkmpx]

@@this is a script to download and setup a woofce local repo ready to be properly synchronized with the remote repos (github) when a user enters commands, it only does the basic stuff...
https://raw.githubusercontent.com/puppylinux-woof-CE/woof-CE/8b60ff5ede7c9adf9d3043a88990e5bce9ebce6a/woof-code/rootfs-skeleton/usr/sbin/gitrepo

@mrfricks @rserwin1