Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How to upgrade os-kernel with source code from kernel.org? #78

Closed
wonleing opened this issue Apr 12, 2021 · 43 comments
Closed

How to upgrade os-kernel with source code from kernel.org? #78

wonleing opened this issue Apr 12, 2021 · 43 comments

Comments

@wonleing
Copy link

wonleing commented Apr 12, 2021

BurmillaOS Version: (ros os version)
v4.14.x
Where are you running BurmillaOS? (docker-machine, AWS, GCE, baremetal, etc.)
KVM, baremetal, iot
Which processor architecture you are using?
x86
Do you use some extra hardware? (GPU, etc)?
no

I am tring to replace 4.14.x kernel with 4.19 kernel, I use os-kernel build a new one and replace the KERNEL_URL in this 'os' project. I succeeded to build the iso, but it can not start properly. The worse thing is I can not see any error log, I don't know how to debug it...

image

Can you give me a proper instruction on how to do it, or some effecient way of debug?

Thank you very much,
Leon

@olljanat
Copy link
Member

You can find our kernel build files from https://github.com/burmilla/os-kernel

However notice that we have made decision that v1.9.x versions will stay on 4.14.x kernels. If you need newer version then I highly recommend that you test our 2.x latest beta version https://github.com/burmilla/os/releases/tag/v2.0.0-beta4

@wonleing
Copy link
Author

I don't quite understand the version policy.
So If I insist to do Kernel 4.19, both v1.9.x and v2.x can support, but v2.x is the better choice. right?

@olljanat
Copy link
Member

olljanat commented Apr 13, 2021

We are trying to use standard software versioning where those numbers indicate about how big change is.

Also we have some targets written to https://github.com/burmilla/os/projects

You can find our original kernel / os versioning discussion from #5

@newhuangchuan
Copy link

Hello, when I use the kernel code in the public network kernel.org to compile, use the public network kernel to compile the .config file to make the kernel, and then the compiled burmillaOS fails to start. May I ask what is in your code? The difference between kurnel-config and the .config file compiled by menuconfg in the kernel code

@olljanat
Copy link
Member

@wonleing @newhuangchuan you can see our kernel configs and all build scripts on os-kernel and from releases view you can find links to exact commits.

However, can you plz explain that what you are trying to achieve by building custom kernel?

@wonleing
Copy link
Author

wonleing commented Apr 20, 2021

@wonleing @newhuangchuan you can see our kernel configs and all build scripts on os-kernel and from releases view you can find links to exact commits.

We compared os-kernel/config/x86/kernel-config and .config that generated by 'make menuconfig' (source code is from kernel.org, same version as describe in os-kernel/config/x86/kernel-config). There are 2000+ differences.

However, can you plz explain that what you are trying to achieve by building custom kernel?

We are maintaining our own kernel source, added patchs drivers and more arches support base on kernel.org kernel.
So enventually we hope to build an iso base on our own kernel source and rootfs.

@olljanat
Copy link
Member

We compared os-kernel/config/x86/kernel-config and .config that generated by 'make menuconfig' (source code is from kernel.org, same version as describe in os-kernel/config/x86/kernel-config). There are 2000+ differences.

ah, true because we have heavily disabled non-used features.

Anyway, I think that setting which you are missing is CONFIG_KERNEL_XZ=y which looks to be disabled by default and which is needed because BurmillaOS uses XZ to compressed base Docker images inside of initrd:

initrd_extract/usr/share/ros$ file *
images-init.tar:   XZ compressed data
images-system.tar: XZ compressed data

We are maintaining our own kernel source, added patchs drivers and more arches support base on kernel.org kernel.
So enventually we hope to build an iso base on our own kernel source and rootfs.

OK. Are those patched kernel sources available somewhere where I can see them? Also do you have patches version of QEMU which can be used to emulate those archs (if they are not included to it by default)?

We have one similar case open on #23 where we noticed that porting BurmillaOS to another archs is quite tricky on current architecture which why I have been doing prototyping about new one on https://github.com/burmilla/os-base-new but currently that one uses rootfs from Debian so if you are looking for archs which are not supported by it that need to be solved first.

@wonleing
Copy link
Author

Are those patched kernel sources available somewhere where I can see them?

unfortunatelly no... it is not controlled by me. And we use real machine instead of QEMU for other arches.

because we have heavily disabled non-used features

So the question is how to generate a new config/x86/kernel-config file when I doing kernel upgrade to 4.19.x or higher? any policy or step or tool? I suppose the 4.14.x kernel-config does not quite suite.

#23 where we noticed that porting BurmillaOS to another archs.

I know the asker of that thread :) Indeed we are trying to do similar thing. And we have already had rootfs of Debian to support that arch.
About system and user docker, can we just use the binary that built by official docker-ce source code (of course added arch support)? Or is there any other tricky change?
We already built out all V19.03.8 docker-ce/docker-ce-cli for all arches.

https://github.com/burmilla/os-base-new

This is awesome. We are trying to use it.

@wonleing
Copy link
Author

@olljanat
Copy link
Member

So the question is how to generate a new config/x86/kernel-config file when I doing kernel upgrade to 4.19.x or higher? any policy or step or tool?

On theory this is only setting which you need to update to default kernel config:

CONFIG_KERNEL_XZ=y

Alternatively you can change this line

ARCHIVE_CMD="xz -4 -e"

to

ARCHIVE_CMD="gzip"

before you generate ISO file. Then standard config from kernel.org should works.

@olljanat
Copy link
Member

@wonleing btw. Is It "LoongArch" support which you are trying to achieve or some other arch?

@wonleing
Copy link
Author

wonleing commented Apr 23, 2021

@wonleing btw. Is It "LoongArch" support which you are trying to achieve or some other arch?

we focus on these arches: amd64, arm64, mips64, loongarch64 and sw64(Alpha)

@wonleing
Copy link
Author

Now we are able to support standard kernel.org 4.19.90 kernel and buildroot-2021.02.1
Can we join as burmillar project maintainer/developer, so we can sync these update to the community?

@olljanat
Copy link
Member

Can we join as burmillar project maintainer/developer, so we can sync these update to the community?

Yes we are looking for maintainers for this project #2 and there is still room for more persons. Also it would be nice if you can share more details about your use case(s) on #6 Especially about that which kind of workloads you are planning to run on top of BurmillaOS after you get in working on all those archs? Also it would be interesting to know that why you want to do it with BurmillaOS instead of some other OS?

@wonleing
Copy link
Author

Also it would be interesting to know that why you want to do it with BurmillaOS instead of some other OS

ContainerOS is much smaller and lighter than common linux distribution while still support containers runtime. So this is paticular fit for VM guestOS and IOT devices when they want to use containerized micro services.
Compare to Fedora CoreOS, BurmillarOS is much smaller and easier to build and maintain. Also I like the idea of 'containerized system process', I think this could be a future trend somehow.

@olljanat
Copy link
Member

olljanat commented Apr 25, 2021

Compare to Fedora CoreOS, BurmillarOS is much smaller and easier to build and maintain.

Agreed, that why I ended up to fork RancherOS.

Also I like the idea of 'containerized system process', I think this could be a future trend somehow.

Yes on theory it is good idea but current system-docker is far from optimal and we need figure out some alternative for it #28 (comment)

@olljanat
Copy link
Member

we focus on these arches: amd64, arm64, mips64, loongarch64 and sw64(Alpha)

@wonleing btw. Which bootloader you use on mips64, loongarch64 and sw64? Is grub available on there or is there need to use some special one?

Also one thing which came to my mind is that you might want to look how our Raspberry Pi build process works as it is probably more near of your use case. For those we build kernel using scrips on https://github.com/burmilla/os-rpi-kernel and actual media gets created by scripts on https://github.com/burmilla/os/tree/master/scripts/images/raspberry-pi-hypriot64

@wonleing
Copy link
Author

Which bootloader you use on mips64, loongarch64 and sw64?

grub. no special things here
For now we mainly build and test burmillaOS on x86_64 KVM. Trying to find the right way to build and upgrade kernel/rootfs/services with our source code (from generic OS source code). Still have many problems to resolve...
hardware related issue could be handled later

@newhuangchuan
Copy link

image
Have you ever encountered os-docker in the system-docker restarting state, and executing the docker command directly shows that it has not been started. Is the docker command not related to os-docker in system-docker?

@olljanat
Copy link
Member

os-docker container run Docker engine/daemon (dockerd). docker command is Docker CLI which connects to it.

Why it is on restarting state can be investigated with sudo system-docker logs docker or looking for "/var/log/docker.log"

@newhuangchuan
Copy link

When looking at /var/log/docker.log, the display is as follows, which seems to be caused by the iptables rules, and iptables cannot create the DOCKER chain of the nat table. But there is only one iptables-xml in the system.
The error is shown in the figure below:
image

@olljanat
Copy link
Member

Which version of code you are using? Error says nf_tables so you are most probably missing at least this one #84

@newhuangchuan
Copy link

The version I use is v2.0.0-beta2.
When checking the rules of iptables, there is only one iptables-xml command that can be used. After installing through apt-get, it is still an error.

@olljanat
Copy link
Member

Hmm. Beta2 is very early draft. It would be better to take beta4 which contains a lot of corrections done after that.

@newhuangchuan
Copy link

Okay, let me switch, thank you very much.

@newhuangchuan
Copy link

Hello, we have used the branch of v1.9.x. Is the firewall he uses by default nf_tables?

@wonleing
Copy link
Author

wonleing commented May 17, 2021

Hello, we have used the branch of v1.9.x. Is the firewall he uses by default nf_tables?

Please merge this patch to your code
bcaa4a1

@newhuangchuan
Copy link

Hello, how do you build user docker and system-docker in BurmillaOS?

@ToeiRei
Copy link

ToeiRei commented May 18, 2021

@newhuangchuan Look at https://github.com/burmilla/os-kernel - the documentation there should help you

@olljanat
Copy link
Member

Hello, how do you build user docker and system-docker in BurmillaOS?

@newhuangchuan user docker uses official static binaries from https://download.docker.com/linux/static/stable/ those as packed inside of docker images using these https://github.com/burmilla/os-services/tree/master/images/10-docker-20.10.5

System-docker sources are on https://github.com/burmilla/os-system-docker and repositories referenced by it but we have not actually ever build it. I just copied binaries from https://github.com/rancher/os-system-docker/releases/tag/17.06-ros6 to our repo.

What I know for sure is that system-docker is based on Docker 17.06 and contain these customizations on engine burmilla/docker@4119920...6368e2f and these on cli burmilla/docker-cli@f6b3234...fa7c1da

Some more info about it can be found from discussion on #28

@newhuangchuan
Copy link

Hello, I would like to ask if you have ever reported a trash error when building os-system-docker here:
trash: error: no such option: -k
However, there is a trash command in the local system. which trash >> /usr/bin/trash

Usage: trash [OPTION]... FILE...

Put files in trash

Options:
  --version            show program's version number and exit
  -h, --help           show this help message and exit
  -d, --directory      ignored (for GNU rm compatibility)
  -f, --force          ignored (for GNU rm compatibility)
  -i, --interactive    ignored (for GNU rm compatibility)
  -r, -R, --recursive  ignored (for GNU rm compatibility)
  -v, --verbose        explain what is being done

To remove a file whose name starts with a '-', for example '-foo',
use one of these commands:

    trash -- -foo

    trash ./-foo

Report bugs to https://github.com/andreafrancia/trash-cli/issues
It seems that there is no -k parameter in the parameters.

There is also whether there is a way to make os-docker mirroring

@newhuangchuan
Copy link

Hello, I have built the os-services project here. May I ask, after the image under this project is built, is it used as a burmillaOS system? For example, starting an nginx service in burmillaOS should be directly accessible like other systems, right? Or must the services project be used for construction?
`Images ready to push:

rancher/os-alpineconsole:f0e2da8-dirty
rancher/os-centosconsole:f0e2da8-dirty
rancher/os-debianconsole:f0e2da8-dirty
rancher/os-fedoraconsole:f0e2da8-dirty
rancher/os-hypervvmtools:f0e2da8-dirty
rancher/os-modemmanager:f0e2da8-dirty
rancher/os-openvmtools:f0e2da8-dirty
rancher/os-pingan:f0e2da8-dirty
rancher/os-qemuguestagent:f0e2da8-dirty
rancher/os-selinuxtools:f0e2da8-dirty
rancher/os-ubuntuconsole:f0e2da8-dirty
rancher/os-waagent:f0e2da8-dirty
rancher/os-amazonmetadata:f0e2da8-dirty
rancher/os-iscsi:f0e2da8-dirty
rancher/os-nvidiadriver:f0e2da8-dirty
rancher/os-volumenetshare:f0e2da8-dirty
rancher/os-zfs:f0e2da8-dirty`

@olljanat
Copy link
Member

Those are Docker images. os-config.tpl.yml defines services included to OS which uses those images like this:

os/os-config.tpl.yml

Lines 112 to 398 in 76b4a14

services:
command-volumes:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: echo
labels:
io.rancher.os.createonly: "true"
io.rancher.os.scope: system
log_driver: json-file
net: none
privileged: true
read_only: true
volumes:
- /usr/bin/ros:/usr/bin/ros:ro
- /usr/bin/system-docker:/usr/bin/system-docker:ro
- /usr/bin/system-docker-runc:/usr/bin/system-docker-runc:ro
system-volumes:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: echo
labels:
io.rancher.os.createonly: "true"
io.rancher.os.scope: system
log_driver: json-file
net: none
privileged: true
read_only: true
volumes:
- /dev:/host/dev
- /etc/docker:/etc/docker
- /etc/hosts:/etc/hosts
- /etc/logrotate.d:/etc/logrotate.d
- /etc/resolv.conf:/etc/resolv.conf
- /etc/ssl/certs/ca-certificates.crt:/etc/ssl/certs/ca-certificates.crt.rancher
- /etc/selinux:/etc/selinux
- /lib/firmware:/lib/firmware
- /lib/modules:/lib/modules
- /run:/run
- /usr/share/ros:/usr/share/ros
- /var/lib/boot2docker:/var/lib/boot2docker
- /var/lib/rancher/cache:/var/lib/rancher/cache
- /var/lib/rancher/conf:/var/lib/rancher/conf
- /var/lib/rancher:/var/lib/rancher
- /var/lib/waagent:/var/lib/waagent
- /var/log:/var/log
- /var/run:/var/run
container-data-volumes:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: echo
labels:
io.rancher.os.createonly: "true"
io.rancher.os.scope: system
log_driver: json-file
net: none
privileged: true
read_only: true
volumes:
- /var/lib/user-docker:/var/lib/docker
- /var/lib/m-user-docker:/var/lib/m-user-docker
user-volumes:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: echo
labels:
io.rancher.os.createonly: "true"
io.rancher.os.scope: system
log_driver: json-file
net: none
privileged: true
read_only: true
volumes:
- /home:/home
- /opt:/opt
- /var/lib/kubelet:/var/lib/kubelet
media-volumes:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: echo
labels:
io.rancher.os.createonly: "true"
io.rancher.os.scope: system
log_driver: json-file
net: none
privileged: true
read_only: true
volumes:
- /media:/media:shared
- /mnt:/mnt:shared
all-volumes:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: echo
labels:
io.rancher.os.createonly: "true"
io.rancher.os.scope: system
io.docker.compose.rebuild: always
log_driver: json-file
net: none
privileged: true
read_only: true
volumes_from:
- container-data-volumes
- command-volumes
- media-volumes
- user-volumes
- system-volumes
{{if eq "amd64" .ARCH -}}
acpid:
image: {{.OS_REPO}}/os-acpid:{{.VERSION}}{{.SUFFIX}}
command: /usr/sbin/acpid -f
labels:
io.rancher.os.scope: system
net: host
uts: host
privileged: true
volumes_from:
- command-volumes
- system-volumes
{{end -}}
cloud-init-execute:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: cloud-init-execute -pre-console
labels:
io.rancher.os.detach: "false"
io.rancher.os.scope: system
io.rancher.os.after: ntp
net: host
uts: host
privileged: true
volumes_from:
- system-volumes
volumes:
- /usr/bin/ros:/usr/bin/ros:ro
console:
image: {{.OS_REPO}}/os-console:{{.VERSION}}{{.SUFFIX}}
command: ros console-init
labels:
io.rancher.os.scope: system
io.rancher.os.after: network
io.docker.compose.rebuild: "false"
io.rancher.os.console: default
environment:
- HTTP_PROXY
- HTTPS_PROXY
- NO_PROXY
net: host
uts: host
pid: host
ipc: host
privileged: true
oom_score_adj: -100
restart: always
volumes_from:
- all-volumes
volumes:
- /media:/media:shared
- /mnt:/mnt:shared
logrotate:
image: {{.OS_REPO}}/os-logrotate:{{.VERSION}}{{.SUFFIX}}
command: /usr/sbin/logrotate -v /etc/logrotate.conf
labels:
io.rancher.os.createonly: "true"
io.rancher.os.scope: system
io.rancher.os.before: system-cron
cron.schedule: "@hourly"
uts: host
net: none
privileged: true
volumes_from:
- command-volumes
- system-volumes
network:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: netconf
labels:
io.rancher.os.scope: system
io.rancher.os.after: udev
io.rancher.os.reloadconfig: "true"
net: host
uts: host
pid: host
privileged: true
volumes_from:
- system-volumes
- command-volumes
volumes:
- /usr/bin/iptables:/sbin/iptables:ro
ntp:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: /bin/start_ntp.sh
labels:
io.rancher.os.scope: system
io.rancher.os.after: network
net: host
uts: host
privileged: true
restart: always
volumes_from:
- command-volumes
- system-volumes
preload-user-images:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: ros preload-images
net: host
labels:
io.rancher.os.detach: "false"
io.rancher.os.scope: system
io.rancher.os.after: console
privileged: true
volumes_from:
- command-volumes
- system-volumes
syslog:
image: {{.OS_REPO}}/os-syslog:{{.VERSION}}{{.SUFFIX}}
command: rsyslogd -n
labels:
io.rancher.os.scope: system
log_driver: json-file
net: host
uts: host
privileged: true
restart: always
volumes_from:
- command-volumes
- system-volumes
system-cron:
{{if eq "amd64" .ARCH -}}
image: burmilla/container-crontab:v0.5.0
{{else -}}
image: burmilla/container-crontab:v0.5.0{{.SUFFIX}}
{{end -}}
labels:
io.rancher.os.scope: system
uts: host
net: none
privileged: true
restart: always
volumes:
- /var/run/system-docker.sock:/var/run/docker.sock
environment:
DOCKER_API_VERSION: "1.22"
udev-cold:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: ros udev-settle
labels:
io.rancher.os.detach: "false"
io.rancher.os.scope: system
net: host
uts: host
privileged: true
volumes_from:
- command-volumes
- system-volumes
udev:
image: {{.OS_REPO}}/os-base:{{.VERSION}}{{.SUFFIX}}
command: udevd
labels:
io.rancher.os.detach: "true"
io.rancher.os.scope: system
io.rancher.os.after: udev-cold
net: host
uts: host
privileged: true
restart: always
volumes_from:
- command-volumes
- system-volumes
docker:
{{if eq "amd64" .ARCH -}}
image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}}
{{else -}}
image: {{.OS_REPO}}/os-docker:{{.USER_DOCKER_VERSION}}{{.SUFFIX}}
{{end -}}
command: ros user-docker
environment:
- HTTP_PROXY
- HTTPS_PROXY
- NO_PROXY
labels:
io.rancher.os.scope: system
io.rancher.os.after: console
net: host
pid: host
ipc: host
uts: host
privileged: true
restart: always
volumes_from:
- all-volumes
volumes:
- /sys:/host/sys
- /var/lib/system-docker:/var/lib/system-docker:shared

That is same syntax than on docker-compose files but that file also contains some other OS configurations.
Extra services which can be enabled with sudo ros service ... are defined on those compose files on os-services repo.
Look: https://burmillaos.org/docs/system-services/custom-system-services/

@newhuangchuan
Copy link

Will the construction of burmillaOS depend on os-services?

@olljanat
Copy link
Member

Only user docker image is used from os-services by default. All other system services are build from https://github.com/burmilla/os/tree/master/images

@newhuangchuan
Copy link

image
Hello, on the arm64 system, burmillaOS has been built, and the selected code branch v2.0.0-beta4 and feat/uefi-support are two. After the build is completed, the display is shown when starting through qemu, you know What caused this situation? Is it caused by the absence of efi files in the ISO file during the ISO building process? I mounted the image on the system and compared it with other ISO files.

@olljanat
Copy link
Member

At least you need duplicate these modifications to arm64 file too e42fef0#diff-f05bf44a1c5840775c97074e1e62efa65149d8f6f0d0654b9b80a1b6b65b52e5

@newhuangchuan
Copy link

Hello, I have modified this part in the v2.0.0-beta4 branch, but the built one still has the problem of UEFI not being able to boot. I doubt it is caused by the absence of files required by UEFI in the image.
this is arm64 ubuntu:
boot casper dists efi install md5sum.txt pool ubuntu ubuntu-ports
this is burmillaOS:
boot rancheros

@olljanat
Copy link
Member

Try with this guide https://github.com/mkinney/myranch/blob/master/readme.MD#create-uefi-bootable-iso-from-rancher-os-iso-from-mac and btw after you figure out needed configuration it would be better to comment it to #8

@wonleing
Copy link
Author

Does 2.0.0-beta4 tag support uefi? I saw an uefi-support branch, we tried it, cannot boot up either.
Another question is: Does arm64 have to use uefi? Anyway to swith it to bios in arm64 KVM?

@olljanat
Copy link
Member

olljanat commented Jun 15, 2021

No 2.0.0-beta4 does not have UEFI support yet. That is biggest reason why it is still marked as beta instead of release candidate. Arm64 can stay with BIOS only unless there is users who need UEFI support for it.

@newhuangchuan
Copy link

If some users need UEFI to boot the system for installation, how to support it?
All burmillaOS ISOs can be installed to bare metal through a USB flash drive, right?

@olljanat
Copy link
Member

Yes all BurmillaOS ISOs can be installed from USB. UEFI installation need special partitioning which is not currently handled by ros so those need to be done manually until UEFI support is implemented to installer.

@olljanat olljanat closed this as not planned Won't fix, can't repro, duplicate, stale Aug 20, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants