Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incorporate GitHub Actions to create new releases #65

Closed
rluzuriaga opened this issue Feb 15, 2023 · 20 comments · Fixed by #68 or #73
Closed

Incorporate GitHub Actions to create new releases #65

rluzuriaga opened this issue Feb 15, 2023 · 20 comments · Fixed by #68 or #73

Comments

@rluzuriaga
Copy link
Member

I think the best way to incorporate GitHub Actions is by having a manual workflow.
The manual workflow would have a few questions to fill out the release information.
Have it look something like this:
Screenshot from 2023-02-14 16-59-16

The workflow would build all kernels and buildinits, then upload them to the release.

@Sebastian-Roth
Copy link
Member

Sounds good to me!

@Sebastian-Roth
Copy link
Member

@rluzuriaga I just stumbled upon an interesting difference when building with GH Actions:

./build.sh -nfa x64
  shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
Checking packages needed for building
 * WARNING: Did not find a patch file building filesystem without patches!
Preparing buildroot 2022.02.9 on x64 build:
 * Downloading buildroot source package........................Done
 * Extracting buildroot sources................................Done
 * Preparing code..............................................mkdir -p /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/lxdialog
PKG_CONFIG_PATH="" make CC="/usr/bin/gcc" HOSTCC="/usr/bin/gcc" \
    obj=/home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config -C support/kconfig -f Makefile.br conf
make[1]: Entering directory '/home/runner/work/fos/fos/fssourcex64/support/kconfig'
/usr/bin/gcc -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -DCURSES_LOC="<ncurses.h>" -DNCURSES_WIDECHAR=1 -DLOCALE  -I/home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"  -MM *.c > /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/.depend 2>/dev/null || :
/usr/bin/gcc -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -DCURSES_LOC="<ncurses.h>" -DNCURSES_WIDECHAR=1 -DLOCALE  -I/home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"   -c conf.c -o /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/conf.o
/usr/bin/gcc -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -DCURSES_LOC="<ncurses.h>" -DNCURSES_WIDECHAR=1 -DLOCALE  -I/home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"  -I. -c /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/zconf.tab.c -o /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/zconf.tab.o
/usr/bin/gcc -D_DEFAULT_SOURCE -D_XOPEN_SOURCE=600 -DCURSES_LOC="<ncurses.h>" -DNCURSES_WIDECHAR=1 -DLOCALE  -I/home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"   /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/conf.o /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/zconf.tab.o  -o /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/conf
rm /home/runner/work/fos/fos/fssourcex64/output/build/buildroot-config/zconf.tab.c
make[1]: Leaving directory '/home/runner/work/fos/fos/fssourcex64/support/kconfig'
Error in reading or end of file.
*
* Restart config...
*
*
* mpd
*
mpd (BR2_PACKAGE_MPD) [N/y/?] (NEW) 
Error in reading or end of file.
Error in reading or end of file.
*
Error in reading or end of file.
* Interpreter languages and scripting
*
4th (BR2_PACKAGE_4TH) [N/y/?] n
enscript (BR2_PACKAGE_ENSCRIPT) [N/y/?] n
erlang (BR2_PACKAGE_ERLANG) [N/y/?] n
execline (BR2_PACKAGE_EXECLINE) [N/y/?] n
ficl (BR2_PACKAGE_FICL) [N/y/?] n
gauche (BR2_PACKAGE_GAUCHE) [N/y/?] n
guile (BR2_PACKAGE_GUILE) [N/y/?] n
haserl (BR2_PACKAGE_HASERL) [N/y/?] n
janet (BR2_PACKAGE_JANET) [N/y/?] n
jimtcl (BR2_PACKAGE_JIMTCL) [N/y/?] n
lua (BR2_PACKAGE_LUA) [N/y/?] n
luajit (BR2_PACKAGE_LUAJIT) [N/y/?] n
micropython (BR2_PACKAGE_MICROPYTHON) [N/y/?] n
moarvm (BR2_PACKAGE_MOARVM) [N/y/?] n
mono (BR2_PACKAGE_MONO) [N/y/?] n
nodejs (BR2_PACKAGE_NODEJS) [N/y/?] (NEW) 
*
* openjdk needs X.Org
*
perl (BR2_PACKAGE_PERL) [N/y/?] n
php (BR2_PACKAGE_PHP) [N/y/?] n
python3 (BR2_PACKAGE_PYTHON3) [N/y/?] n
quickjs (BR2_PACKAGE_QUICKJS) [N/y/?] n
ruby (BR2_PACKAGE_RUBY) [N/y/?] n
tcl (BR2_PACKAGE_TCL) [N/y/?] n
*
* Host utilities
*
host abootimg (BR2_PACKAGE_HOST_ABOOTIMG) [N/y/?] n
host aespipe (BR2_PACKAGE_HOST_AESPIPE) [N/y/?] n
host android-tools (BR2_PACKAGE_HOST_ANDROID_TOOLS) [N/y/?] n
host asn1c (BR2_PACKAGE_HOST_ASN1C) [N/y/?] n
host babeltrace2 (BR2_PACKAGE_HOST_BABELTRACE2) [N/y/?] n
host bmap-tools (BR2_PACKAGE_HOST_BMAP_TOOLS) [N/y/?] n
host btrfs-progs (BR2_PACKAGE_HOST_BTRFS_PROGS) [N/y/?] n
host checkpolicy (BR2_PACKAGE_HOST_CHECKPOLICY) [N/y/?] n
host checksec (BR2_PACKAGE_HOST_CHECKSEC) [N/y/?] n
host cmake (BR2_PACKAGE_HOST_CMAKE) [N/y/?] n
host cramfs (BR2_PACKAGE_HOST_CRAMFS) [N/y/?] n
host cryptsetup (BR2_PACKAGE_HOST_CRYPTSETUP) [N/y/?] n
host dbus-python (BR2_PACKAGE_HOST_DBUS_PYTHON) [N/y/?] n
host delve (BR2_PACKAGE_HOST_DELVE) [N/y/?] n
host dfu-util (BR2_PACKAGE_HOST_DFU_UTIL) [N/y/?] n
host dos2unix (BR2_PACKAGE_HOST_DOS2UNIX) [N/y/?] n
host dosfstools (BR2_PACKAGE_HOST_DOSFSTOOLS) [N/y/?] n
host doxygen (BR2_PACKAGE_HOST_DOXYGEN) [N/y/?] n
host dtc (BR2_PACKAGE_HOST_DTC) [N/y/?] n
host e2fsprogs (BR2_PACKAGE_HOST_E2FSPROGS) [Y/?] y
host e2tools (BR2_PACKAGE_HOST_E2TOOLS) [N/y/?] n
host environment-setup (BR2_PACKAGE_HOST_ENVIRONMENT_SETUP) [N/y/?] n
host erofs-utils (BR2_PACKAGE_HOST_EROFS_UTILS) [N/y/?] n
host exfatprogs (BR2_PACKAGE_HOST_EXFATPROGS) [N/y/?] n
host f2fs-tools (BR2_PACKAGE_HOST_F2FS_TOOLS) [N/y/?] n
host faketime (BR2_PACKAGE_HOST_FAKETIME) [N/y/?] n
host fatcat (BR2_PACKAGE_HOST_FATCAT) [N/y/?] n
host firmware-utils (BR2_PACKAGE_HOST_FIRMWARE_UTILS) [N/y/?] n
host fwup (BR2_PACKAGE_HOST_FWUP) [N/y/?] n
host genext2fs (BR2_PACKAGE_HOST_GENEXT2FS) [Y/n/?] y
host genimage (BR2_PACKAGE_HOST_GENIMAGE) [N/y/?] n
host genpart (BR2_PACKAGE_HOST_GENPART) [N/y/?] n
host gnupg (BR2_PACKAGE_HOST_GNUPG) [N/y/?] n
host gptfdisk (BR2_PACKAGE_HOST_GPTFDISK) [Y/n/?] y
host imagemagick (BR2_PACKAGE_HOST_IMAGEMAGICK) [N/y/?] n
host imx-mkimage (BR2_PACKAGE_HOST_IMX_MKIMAGE) [N/y/?] n
host jq (BR2_PACKAGE_HOST_JQ) [Y/n/?] y
host jsmin (BR2_PACKAGE_HOST_JSMIN) [N/y/?] n
host kmod (BR2_PACKAGE_HOST_KMOD) [N/y/?] n
host libp11 (BR2_PACKAGE_HOST_LIBP11) [N/y/?] n
host lld (BR2_PACKAGE_HOST_LLD) [N/y/?] n
host lpc3250loader (BR2_PACKAGE_HOST_LPC3250LOADER) [N/y/?] n
host lttng-babeltrace (BR2_PACKAGE_HOST_LTTNG_BABELTRACE) [N/y/?] n
host mender-artifact (BR2_PACKAGE_HOST_MENDER_ARTIFACT) [N/y/?] n
host meson-tools (BR2_PACKAGE_HOST_MESON_TOOLS) [N/y/?] n
host mkpasswd (BR2_PACKAGE_HOST_MKPASSWD) [N/y/?] n
host mtd, jffs2 and ubi/ubifs tools (BR2_PACKAGE_HOST_MTD) [N/y/?] n
host mtools (BR2_PACKAGE_HOST_MTOOLS) [N/y/?] n
host nodejs (BR2_PACKAGE_HOST_NODEJS) [N/y/?] (NEW) 
host-odb (BR2_PACKAGE_HOST_ODB) [N/y/?] n
host openocd (BR2_PACKAGE_HOST_OPENOCD) [N/y/?] n
host opkg-utils (BR2_PACKAGE_HOST_OPKG_UTILS) [N/y/?] n
host pahole (BR2_PACKAGE_HOST_PAHOLE) [N/y/?] n
host parted (BR2_PACKAGE_HOST_PARTED) [N/y/?] n
host pigz (BR2_PACKAGE_HOST_PIGZ) [N/y/?] n
Wed Feb 22 20:41:10 UTC 2023 - building ...
Wed Feb 22 20:41:40 UTC 2023 - building ...
Wed Feb 22 20:42:10 UTC 2023 - building ...
Wed Feb 22 20:42:40 UTC 2023 - building ...
...

But when manually building from the latest I see this:

./build.sh -nfa x64
Checking packages needed for building
 * WARNING: Did not find a patch file building filesystem without patches!
Preparing buildroot 2022.02.9 on x64 build:
 * Extracting buildroot sources................................Done
 * Preparing code..............................................mkdir -p /home/sebastian/fos/fssourcex64/output/build/buildroot-config/lxdialog
PKG_CONFIG_PATH="" make CC="/usr/bin/gcc" HOSTCC="/usr/bin/gcc" \
    obj=/home/sebastian/fos/fssourcex64/output/build/buildroot-config -C support/kconfig -f Makefile.br conf
make[1]: Entering directory '/home/sebastian/fos/fssourcex64/support/kconfig'
/usr/bin/gcc -DCURSES_LOC="<ncurses.h>" -DLOCALE  -I/home/sebastian/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"  -MM *.c > /home/sebastian/fos/fssourcex64/output/build/buildroot-config/.depend 2>/dev/null || :
/usr/bin/gcc -DCURSES_LOC="<ncurses.h>" -DLOCALE  -I/home/sebastian/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"   -c conf.c -o /home/sebastian/fos/fssourcex64/output/build/buildroot-config/conf.o
/usr/bin/gcc -DCURSES_LOC="<ncurses.h>" -DLOCALE  -I/home/sebastian/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"  -I. -c /home/sebastian/fos/fssourcex64/output/build/buildroot-config/zconf.tab.c -o /home/sebastian/fos/fssourcex64/output/build/buildroot-config/zconf.tab.o
/usr/bin/gcc -DCURSES_LOC="<ncurses.h>" -DLOCALE  -I/home/sebastian/fos/fssourcex64/output/build/buildroot-config -DCONFIG_=\"\"   /home/sebastian/fos/fssourcex64/output/build/buildroot-config/conf.o /home/sebastian/fos/fssourcex64/output/build/buildroot-config/zconf.tab.o  -o /home/sebastian/fos/fssourcex64/output/build/buildroot-config/conf
rm /home/sebastian/fos/fssourcex64/output/build/buildroot-config/zconf.tab.c
make[1]: Leaving directory '/home/sebastian/fos/fssourcex64/support/kconfig'
#
# configuration written to /home/sebastian/fos/fssourcex64/.config
#
Done
Wed Feb 22 15:19:23 CST 2023 - building ...
Wed Feb 22 15:19:53 CST 2023 - building ...
Wed Feb 22 15:20:23 CST 2023 - building ...
Wed Feb 22 15:20:53 CST 2023 - building ...
...

Looks like it's unable to use our existing configs....

@Sebastian-Roth
Copy link
Member

Same with the kernels:

./build.sh -nka x64
Checking packages needed for building
Preparing kernel 5.15.93 on x64 build:
 * Downloading kernel source...................................Done
 * Extracting kernel source....................................Done
 * Preparing kernel source.....................................Done
 * WARNING: Did not find a patch file building vanilla kernel without patches!
 * Cloning Linux firmware repository...........................Done
  HOSTCC  scripts/basic/fixdep
  HOSTCC  scripts/kconfig/conf.o
  HOSTCC  scripts/kconfig/confdata.o
  HOSTCC  scripts/kconfig/expr.o
  LEX     scripts/kconfig/lexer.lex.c
  YACC    scripts/kconfig/parser.tab.[ch]
  HOSTCC  scripts/kconfig/lexer.lex.o
  HOSTCC  scripts/kconfig/menu.o
  HOSTCC  scripts/kconfig/parser.tab.o
  HOSTCC  scripts/kconfig/preprocess.o
  HOSTCC  scripts/kconfig/symbol.o
  HOSTCC  scripts/kconfig/util.o
  HOSTLD  scripts/kconfig/conf

*
* Restart config...
*
*
* Mitigations for speculative execution vulnerabilities
*
...

@rluzuriaga
Copy link
Member Author

rluzuriaga commented Feb 22, 2023

@Sebastian-Roth That's odd. When I run it manually I get the exact same output as GitHub Actions. I just thought that was the normal process.
How are you running it manually?

@Sebastian-Roth
Copy link
Member

@rluzuriaga said:

How are you running it manually?

Exactly as seen above: ./build.sh -nfa x64 - this is on Debian Linux

@rluzuriaga
Copy link
Member Author

@Sebastian-Roth Oh maybe it's an Ubuntu thing. I'll look into it.
From my testing, I've been building on Ubuntu without any issues and the kernels and inits all work fine.

@Sebastian-Roth
Copy link
Member

I am just worried about the "Restarting config" message and I guess the "Error in reading or end of file" is part of that as well. I am not exactly sure but I guess this happens when you run make oldconfig but the .config file is missing or corrupt. But on the other hand you'd think that a default kernel config would never work to boot up a FOS machine...

@rluzuriaga
Copy link
Member Author

rluzuriaga commented Feb 23, 2023

@Sebastian-Roth So I created a brand-new Debian 11 VM to test the difference in building between it and Ubuntu.
I get the exact same thing on Debian and Ubuntu.
I'm wondering if you have any special setup on your machine.

Screenshot 2023-02-22 170806
Don't mind the characters at the end, that was an accident.

@rluzuriaga
Copy link
Member Author

@Sebastian-Roth Okay I did some more testing and the same appeared on Debian 11 and 10. I tried Debian 9 and then I get the same output as your local machine. Is the machine running Debian 9?

@Sebastian-Roth
Copy link
Member

@rluzuriaga Right, got me there! Using an old Debian 9 system for building. I will test on an up to date system soon. I think we can just leave this topic open for the time being. I will get back to it when I have more time.

Thanks heaps for your testing!

@rluzuriaga
Copy link
Member Author

@Sebastian-Roth I think I know why Debian 9 doesn't give that "issue". When building the kernels and inits on Debian 9 or Ubuntu 18.04 and bellow, it doesn't ask for the following packages. On newer versions of Debian and Ubuntu we would need to add these 4 lines to the filesystem config files.

# BR2_PACKAGE_MPD is not set
# BR2_PACKAGE_NODEJS is not set
# BR2_PACKAGE_HOST_NODEJS is not set
# BR2_PACKAGE_HOST_QEMU is not set

MPD (Music Player Daemon) shouldn't be needed neither should nodeJS or QEMU.


These are the lines that need to be added to the kernel configs.

x64 Kernel Config

# CONFIG_SLS is not set
# CONFIG_ZERO_CALL_USED_REGS is not set
# CONFIG_KCSAN is not set
# CONFIG_KASAN is not set

x86 Kernel Config

CONFIG_STACKPROTECTOR=y
CONFIG_STACKPROTECTOR_STRONG=y
# CONFIG_ZERO_CALL_USED_REGS is not set

arm64 Kernel Config

CONFIG_ARM64_PTR_AUTH_KERNEL=y
CONFIG_ARM64_TLB_RANGE=y
CONFIG_ARM64_MTE=y
# CONFIG_ZERO_CALL_USED_REGS is not set
# CONFIG_KASAN is not set

How did I get these configs? It's just the default that the builder gives.

@Sebastian-Roth
Copy link
Member

@rluzuriaga I just updated the kernel and buildroot configs to build on Debian 10 without those messages (9632dcc). We should definitely be aware of those changes (mentioned by you as well):

Stack Protector buffer overflow detection (STACKPROTECTOR) [Y/n/?] (NEW) 
  Strong Stack Protector (STACKPROTECTOR_STRONG) [Y/n/?] (NEW) 

I think it's ok to enable those but we need to keep that in mind in case people complain about issues with the 32 bit kernel.

@Sebastian-Roth
Copy link
Member

Sebastian-Roth commented Mar 5, 2023

@rluzuriaga To be sure we don't miss this kind of config changes we need to make sure we are using the same OS and version when preparing the configs as the Github Actions. I am wondering if it's wise to set runs-on (refernce) to a specific version so we are in control instead of GH Actions switching to a newer Ubuntu version without us noticing.

Currently ubuntu-22.04 makes sense I think (full list of options).

@rluzuriaga
Copy link
Member Author

rluzuriaga commented Mar 7, 2023

@Sebastian-Roth I agree, I'll change it to 22.04. I think that as soon as a new LTS version comes out, we should try building on it so that we don't fall behind.

Since I'll be making this change, I could make the changes so that the Kernel and Buildroot versions are automatically retrieved from build.sh and we don't have to enter it manually. Also, I can make the release name the same as how they show up now: Latest from MM.DD.YYY and the tag as YYYYMMDD without asking for that input.

@rluzuriaga
Copy link
Member Author

@Sebastian-Roth I also saw this commit that you made to release.sh. I can make another workflow that does that too, or I can modify the current workflow to have a checkbox to make an official release build.
What do you think?

@Sebastian-Roth
Copy link
Member

@rluzuriaga said:

Since I'll be making this change, I could make the changes so that the Kernel and Buildroot versions are automatically retrieved from build.sh and we don't have to enter it manually. Also, I can make the release name the same as how they show up now: Latest from MM.DD.YYY and the tag as YYYYMMDD without asking for that input.

Yeah, that'd be great. I can't see why we would need manually enter that information. If it needs adjustment at some point (e.g. we want other tag names), then we can simply change the YAML again. Great!

I also saw this commit that you made to release.sh. I can make another workflow that does that too, or I can modify the current workflow to have a checkbox to make an official release build. What do you think?

Sounds great as well. I think adding a checkbox (default unchecked) and an extra field named "Release Version" would work perfectly fine.

@Sebastian-Roth
Copy link
Member

@rluzuriaga As well you can remove the "Pre-release" checkbox. While it can be useful I think it can confuse people as well now that we add the "Official release" checkbox. On the Github fos releases page we can manually edit a release and change it to pre-release if wanted. The only case this is playing a role is, that pre-releases are hidden from the FOG web UI kernel update view.

@rluzuriaga
Copy link
Member Author

rluzuriaga commented Mar 14, 2023

@Sebastian-Roth Sounds good to me. Maybe it is a good idea to change the release name to full date. Instead of MM.DD.YYYY it could be MONTH DD, YYYY?

For the official release checkbox, I would have to add a text field for the release name/FOG version, since there is nowhere in the code to get that info.

@lukebarone
Copy link
Member

lukebarone commented Mar 14, 2023 via email

@Sebastian-Roth
Copy link
Member

@rluzuriaga said:

Maybe it is a good idea to change the release name to full date. Instead of MM.DD.YYYY it could be MONTH DD, YYYY?

I side with the format @lukebarone suggested, YYYY-MM-DD. This makes it easy to recognize as a date but is still sortable. While we don't need those sorted yet I am sure there will be a situation in the future where this is going to be handy.

For the official release checkbox, I would have to add a text field for the release name/FOG version, since there is nowhere in the code to get that info.

Yes, would need an extra text field.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants