Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for Zen 3 march #59

Closed
NotOles opened this issue Mar 2, 2021 · 20 comments
Closed

Add support for Zen 3 march #59

NotOles opened this issue Mar 2, 2021 · 20 comments
Labels
enhancement New feature or request wontfix This will not be worked on

Comments

@NotOles
Copy link

NotOles commented Mar 2, 2021

Hi Fabian!

Thanks for this project which has been very helpful to me!

Your Zen 2 kernel seems to work fine with Proxmox on my Zen 3 5950X, but I wonder if you are planning to add Zen 3.

Thanks again so much!

Best wishes and kindest regards,

Tom

@fabianishere fabianishere added the question Further information is requested label Mar 2, 2021
@fabianishere fabianishere changed the title Zen 3 Add support for Zen 3 march Mar 2, 2021
@fabianishere
Copy link
Owner

Hi Tom,

Thank you for your message! Given that I don't have any Zen 3 machines in my cluster (yet), I haven't thought about adding a build for it. Note that the Zen 2 kernel is identical to the generic kernel, except that it is tuned for the Zen 2 micro-architecture (via GCC's -march=znver2 option)

As for adding Zen 3 builds, it seems that support for Zen 3 optimizations has only landed in GCC 11, which has not been released in Ubuntu as far as I know. Though, it should not matter much, using the generic or Zen 2 kernel should work fine for you.

@fabianishere fabianishere added enhancement New feature or request and removed question Further information is requested labels Mar 2, 2021
@NotOles
Copy link
Author

NotOles commented Mar 2, 2021

I don't have any Zen 3 machines in my cluster (yet),

Not trying to push, but, you are more than welcome on my machine.

Zen 3 optimizations has only landed in GCC 11

Thanks for the tip! I didn't know. But I might be okay on compiling edge gcc and its edgy friends. :)

Note that the Zen 2 kernel is identical to the generic kernel, except that it is tuned for the Zen 2 micro-architecture (via GCC's -march=znver2 option)

Yeah, I guessed this. I imagine Zen 3 also might be a compiler flag, simple enough to be within my pay grade. :) I might even try it! :)

It's nice that you make edge kernels more easily available to Proxmox peeps. Thank you again, Fabian!

@fabianishere
Copy link
Owner

Thank you for the nice comment!

If you want to try it yourself, and manage to install GCC 11, I expect the following command to achieve what you want (given that you have cloned the repository):

make PVE_BUILD_CC=gcc-11 PVE_BUILD_CFLAGS="-march=znver3"

@Ramalama2
Copy link

Hey guys, im interested in zen3 either 😂

What are you using as build environment @fabianishere ?

Cause I don't think, we can use anything, it needs to have gcc11 + dwarf from debian testing etc, not that super easy i think 🙈

If i can get it working, i can probably help out building and uploading it.

And the kernels got gigantic, holy fuck, i remember 15y back, how small they were, whatever, offtopic 😂

So please provide some info for a good build environment.
And another thing is, do you use menuconfig and set anything when you build these? Cause i didn't seen that any is provided.

Additionally, how it works now with openzfs? This needs to be cloned inside the kernel somehow either?
Or recompiled afterwards on the system?

Sorry, im a bit outdated 🤦‍♂️

@fabianishere
Copy link
Owner

@Ramalama2

You might try to install GCC 11 from Debian Experimental. You can then just follow the steps in the README to obtain the repository and build the kernel as follows:

make PVE_BUILD_CC=gcc-11 PVE_BUILD_CFLAGS="-march=znver3"

This will also build OpenZFS for you and include it in the kernel package.

The configuration for these kernels is based on the original Proxmox configs (which is based in turn on Ubuntu). See

PVE_CONFIG_OPTS= \
-m INTEL_MEI_WDT \
-d CONFIG_SND_PCM_OSS \
-e CONFIG_TRANSPARENT_HUGEPAGE_MADVISE \
-d CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS \
-m CONFIG_CEPH_FS \
-m CONFIG_BLK_DEV_NBD \
-m CONFIG_BLK_DEV_RBD \
-d CONFIG_SND_PCSP \
-m CONFIG_BCACHE \
-m CONFIG_JFS_FS \
-m CONFIG_HFS_FS \
-m CONFIG_HFSPLUS_FS \
-e CIFS_SMB_DIRECT \
-e CONFIG_BRIDGE \
-e CONFIG_BRIDGE_NETFILTER \
-e CONFIG_BLK_DEV_SD \
-e CONFIG_BLK_DEV_SR \
-e CONFIG_BLK_DEV_DM \
-e CONFIG_BLK_DEV_NVME \
-e CONFIG_NLS_ISO8859_1 \
-d CONFIG_INPUT_EVBUG \
-d CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND \
-e CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE \
-d CONFIG_MODULE_SIG \
-d CONFIG_MEMCG_DISABLED \
-e CONFIG_MEMCG_SWAP_ENABLED \
-e CONFIG_HYPERV \
-m CONFIG_VFIO_IOMMU_TYPE1 \
-m CONFIG_VFIO_VIRQFD \
-m CONFIG_VFIO \
-m CONFIG_VFIO_PCI \
-m CONFIG_USB_XHCI_HCD \
-m CONFIG_USB_XHCI_PCI \
-m CONFIG_USB_EHCI_HCD \
-m CONFIG_USB_EHCI_PCI \
-m CONFIG_USB_EHCI_HCD_PLATFORM \
-m CONFIG_USB_OHCI_HCD \
-m CONFIG_USB_OHCI_HCD_PCI \
-m CONFIG_USB_OHCI_HCD_PLATFORM \
-d CONFIG_USB_OHCI_HCD_SSB \
-m CONFIG_USB_UHCI_HCD \
-d CONFIG_USB_SL811_HCD_ISO \
-e CONFIG_MEMCG_KMEM \
-d CONFIG_DEFAULT_CFQ \
-e CONFIG_DEFAULT_DEADLINE \
-e CONFIG_MODVERSIONS \
-d CONFIG_DEFAULT_SECURITY_DAC \
-e CONFIG_DEFAULT_SECURITY_APPARMOR \
--set-str CONFIG_DEFAULT_SECURITY apparmor \
-d CONFIG_UNWINDER_ORC \
-d CONFIG_UNWINDER_GUESS \
-e CONFIG_UNWINDER_FRAME_POINTER \
-d CONFIG_SECURITY_LOCKDOWN_LSM \
-d CONFIG_SECURITY_LOCKDOWN_LSM_EARLY \
--set-str CONFIG_LSM yama,integrity,apparmor \
-e CONFIG_PAGE_TABLE_ISOLATION

As for my build environment, I use a Ubuntu Focal-based 64-core system for the Github Actions Runner.

@Ramalama2
Copy link

Thanks, im going to try out after work!

@Ramalama2
Copy link

Ramalama2 commented Mar 11, 2021

Hi, just a small update,

Im compiling 5.11.5 right now, but didn't used -j`nproc --all`, so it will take forever xD
But i used the -O3 flag additionally.
It looks quite good with gcc-11, so far no problems. I using debian bullseye with the experimental repo as environment.

However, are there ways to improve the rules somehow?
because there are so many unnecessary drivers that blows the kernel up, that no one needs on proxmox or a hypervisor.
like 80211 for example and a ton of gpu drivers, i mean even nvidia & amd is unneccessary, because 99,9% of users will pass them through and blacklist those xD

Just looking for ways to slim down the kernel and maybe optimize it more for zen3.

Another thing is, we don't need some intel drivers on an amd kernel & opposite.

However, when the compilation is done, i will send a link here, for anyone interrested.
Cheers

@fabianishere
Copy link
Owner

fabianishere commented Mar 11, 2021

You can pass your own kernel configuration file if you prefer, but this requires some changes. To change the Proxmox-specific options, modify:

PVE_CONFIG_OPTS= \
-m INTEL_MEI_WDT \
-d CONFIG_SND_PCM_OSS \
-e CONFIG_TRANSPARENT_HUGEPAGE_MADVISE \
-d CONFIG_TRANSPARENT_HUGEPAGE_ALWAYS \
-m CONFIG_CEPH_FS \
-m CONFIG_BLK_DEV_NBD \
-m CONFIG_BLK_DEV_RBD \
-d CONFIG_SND_PCSP \
-m CONFIG_BCACHE \
-m CONFIG_JFS_FS \
-m CONFIG_HFS_FS \
-m CONFIG_HFSPLUS_FS \
-e CIFS_SMB_DIRECT \
-e CONFIG_BRIDGE \
-e CONFIG_BRIDGE_NETFILTER \
-e CONFIG_BLK_DEV_SD \
-e CONFIG_BLK_DEV_SR \
-e CONFIG_BLK_DEV_DM \
-e CONFIG_BLK_DEV_NVME \
-e CONFIG_NLS_ISO8859_1 \
-d CONFIG_INPUT_EVBUG \
-d CONFIG_CPU_FREQ_DEFAULT_GOV_ONDEMAND \
-e CONFIG_CPU_FREQ_DEFAULT_GOV_PERFORMANCE \
-d CONFIG_MODULE_SIG \
-d CONFIG_MEMCG_DISABLED \
-e CONFIG_MEMCG_SWAP_ENABLED \
-e CONFIG_HYPERV \
-m CONFIG_VFIO_IOMMU_TYPE1 \
-m CONFIG_VFIO_VIRQFD \
-m CONFIG_VFIO \
-m CONFIG_VFIO_PCI \
-m CONFIG_USB_XHCI_HCD \
-m CONFIG_USB_XHCI_PCI \
-m CONFIG_USB_EHCI_HCD \
-m CONFIG_USB_EHCI_PCI \
-m CONFIG_USB_EHCI_HCD_PLATFORM \
-m CONFIG_USB_OHCI_HCD \
-m CONFIG_USB_OHCI_HCD_PCI \
-m CONFIG_USB_OHCI_HCD_PLATFORM \
-d CONFIG_USB_OHCI_HCD_SSB \
-m CONFIG_USB_UHCI_HCD \
-d CONFIG_USB_SL811_HCD_ISO \
-e CONFIG_MEMCG_KMEM \
-d CONFIG_DEFAULT_CFQ \
-e CONFIG_DEFAULT_DEADLINE \
-e CONFIG_MODVERSIONS \
-d CONFIG_DEFAULT_SECURITY_DAC \
-e CONFIG_DEFAULT_SECURITY_APPARMOR \
--set-str CONFIG_DEFAULT_SECURITY apparmor \
-d CONFIG_UNWINDER_ORC \
-d CONFIG_UNWINDER_GUESS \
-e CONFIG_UNWINDER_FRAME_POINTER \
-d CONFIG_SECURITY_LOCKDOWN_LSM \
-d CONFIG_SECURITY_LOCKDOWN_LSM_EARLY \
--set-str CONFIG_LSM yama,integrity,apparmor \
-e CONFIG_PAGE_TABLE_ISOLATION

If you also want to remove the Ubuntu/Debian configuration options, modify:

cat ${BUILD_DIR}/${KERNEL_SRC}/debian.master/config/config.common.ubuntu ${BUILD_DIR}/${KERNEL_SRC}/debian.master/config/${ARCH}/config.common.${ARCH} ${BUILD_DIR}/${KERNEL_SRC}/debian.master/config/${ARCH}/config.flavour.generic > ${KERNEL_CFG_ORG}

@Ramalama2
Copy link

Ramalama2 commented Mar 11, 2021

I simply added this into pve-edge-kernel/debian/rules into the PVE_CONFIG_OPTS category xD

-d CONFIG_DRM_RADEON \
-d CONFIG_DRM_AMDGPU \
-d CONFIG_DRM_NOUVEAU \
-d CONFIG_MICROCODE_INTEL \
-d CONFIG_NFC \
-d CONFIG_MAC80211 \
-d CONFIG_CFG80211 \
-d CONFIG_BT \
-d CONFIG_RTL8192U \
-d CONFIG_RTLLIB \
-d CONFIG_WIMAX \
-d CONFIG_ANDROID \

Looked like the easiest way for me to remove the bloat, those things are self explaining, tryed to remove Wireless/BT/Intel/GPU driver for amd based kernel on a hypervisor.


However, there are more things that needs to get removed, everyone that want to help making a list is greatly welcome:

isdn
misdn
thermal/intel
media/radio
media/cec
media/dvb
media/usb/dvb
media/tuners
drivers/input/touchscreen
drivers/input/tablet
drivers/input/joystick
drivers/firewire
drivers/net/ieee802154
drivers/net/ethernet/qualcomm
drivers/net/ethernet/huawei
drivers/net/ethernet/google
drivers/net/ethernet/amazon
drivers/net/ethernet/3com
drivers/net/dsa/hirschmann
drivers/misc/cardreader
drivers/gpu/drm/i915 - gvt-g (but not available on amd systems anyway, until iris-xe pci card & until gvt-g drivers support those)
drivers/video/backlight
arch/x86/xen
arch/x86/ia32 - needed?
drivers/staging/media/atomisp
drivers/platform/chrome
drivers/platform/surface
drivers/pcmcia
drivers/iio - what is this?
net/mac802154
net/wireless
sound/xen
sound/soc/intel
sound/firewire
sound/pci - is there anything that is needed?
sound/soc/intel

This looks like a random list, but i think no one needs this in the zen kernel.
There is for sure a ton more, but i don't have that much clue and need some help xD

However, here is the kernel (without these in the kernel: amd/radeon/nouveau/bt/wifi etc):
https://cloud.golima.de/s/HJs582PrX4cpTza
(kernel+headers+libc)
EDIT: Don't use this, some packages failed, needs a recompile, i will update this. See below, i missed some packages!

Makefile.config:555: No sys/sdt.h found, no SDT events are defined, please install systemtap-sdt-devel or systemtap-sdt-dev
Makefile.config:603: No libunwind found. Please install libunwind-dev[el] >= 1.1 and/or set LIBUNWIND_DIR
Makefile.config:782: Python support disabled by user
Makefile.config:833: Old version of libbfd/binutils things like PE executable profiling will not be available
Makefile.config:894: No libzstd found, disables trace compression, please install libzstd-dev[el] and/or set LIBZSTD_DIR
Makefile.config:905: No libcap found, disables capability support, please install libcap-devel/libcap-dev
Makefile.config:973: No libbabeltrace found, disables 'perf data' CTF format support, please install libbabeltrace-dev[el]/libbabeltrace-ctf-dev
Makefile.config:999: No alternatives command found, you need to set JDIR= to point to the root of your Java directory

Auto-detecting system features:
...                         dwarf: [ on  ]
...            dwarf_getlocations: [ on  ]
...                         glibc: [ on  ]
...                        libbfd: [ OFF ]
...                libbfd-buildid: [ OFF ]
...                        libcap: [ OFF ]
...                        libelf: [ on  ]
...                       libnuma: [ on  ]
...        numa_num_possible_cpus: [ on  ]
...                       libperl: [ OFF ]
...                     libpython: [ OFF ]
...                     libcrypto: [ on  ]
...                     libunwind: [ OFF ]
...            libdw-dwarf-unwind: [ on  ]
...                          zlib: [ on  ]
...                          lzma: [ on  ]
...                     get_cpuid: [ on  ]
...                           bpf: [ on  ]
...                        libaio: [ on  ]
...                       libzstd: [ OFF ]
...        disassembler-four-args: [ OFF ]

That means for me, that we need some additional packages:
apt install systemtap-sdt-dev libunwind-dev libzstd-dev libcap-dev libbabeltrace-dev libpython3.9-dev libperl-dev binutils-dev

(i have installed default-jdk either, dunno what the compiler wants, but whatever, enough diskspace is available)

Will Provide an updated Package in a bit.

@Ramalama2
Copy link

Ramalama2 commented Mar 11, 2021

Unbenannt2
Okay, after removing binutils-dev, it works....
Forgot to link python-config... Or didn't knew before... fixed now anyway too xD

but this here doesnt work;
asd

And that feels somehow super wrong.

How does that section looks in your compiler?

@Ramalama2
Copy link

https://lists.debian.org/debian-release/2015/02/msg00080.html

Well i found this, seems like it's intended.
But it doesn't make any sense, why the hell they provide libbfd if it has an incompatible license at all?

That means that everything is right?

@Ramalama2
Copy link

Okay here we go:
Build env: Debian Bullseye + Experimental repo (gcc-11 + other packages)
Compilation Flags: make PVE_BUILD_CC=gcc-11 PVE_BUILD_CFLAGS="-O3 -march=znver3"
Kernel: 5.11.5 + openzfs 2.0.3
Download: https://cloud.golima.de/s/jMHp4poNJ3Dq27C

Removed:

-d CONFIG_XEN \
-d CONFIG_DRM_RADEON \
-d CONFIG_DRM_AMDGPU \
-d CONFIG_DRM_NOUVEAU \
-d CONFIG_DRM_I915 \
-d CONFIG_DRM_XEN \
-d CONFIG_DRM_GMA500 \
-d CONFIG_SND_FIREWIRE \
-d CONFIG_SND_XEN_FRONTEND \
-d CONFIG_HYPERV \
-d CONFIG_SURFACE_PLATFORMS \
-d CONFIG_NET_VENDOR_GOOGLE \
-d CONFIG_CHROME_PLATFORMS \
-d CONFIG_MICROCODE_INTEL \
-d CONFIG_NFC \
-d CONFIG_MAC80211 \
-d CONFIG_CFG80211 \
-d CONFIG_BT \
-d CONFIG_RTL8192U \
-d CONFIG_RTLLIB \
-d CONFIG_WIMAX \
-d CONFIG_ANDROID \

Why removed? -> Because it is an amd zen3 hypervisor, doesn't need intel and doesn't need GPU/Sound/Wifi/BT etc...

Cheers

@Ramalama2
Copy link

Ramalama2 commented Mar 12, 2021

Okay, i tested what i compiled 🤣

FML, this has less todo with the compilation, and more with the fact of 5.11 itself.
But all my errors are gone!

  • ethtool let you set fixed 2,5GB/s speed (normally it works only through autoneg, but autoneg at this weird 2,5G standard is slow af) (this isn't even possible with mainline intel driver, but with 5.11 it is xD)
  • RDRAND Zen3 issues with zen3 (systemd) are fixed!!!! This driven me crazy xD
  • EDAC ECC, is fully working!

FreeBSD 12.2 (OpenSense) doesn't boots anymore with cpu "host" -> Setted to EPYC and it boots...
Slowly im getting mad of FreeBSD, so many bugs, that doesn't have even todo with new hardware, like q35 passthrough...

However, everything works perfectly!!! Im happy that i did this!
Thanks for help @fabianishere

I can try to provide the builds from time to time!

@Ramalama2
Copy link

Hey guys, small update, as next 5.11.7 is comming, since it has more changes as the previous ones:

Changelog 5.11.7:
- iommu/amd: Fix performance counter initialization
- Full Changelog (305 Changes): https://lwn.net/Articles/849642/

Changelog 5.11.6:
- Full Changelog (37 Changes): https://lwn.net/Articles/849101/

Changelog 5.11.5:
- Full Changelog (45 Changes): https://lwn.net/Articles/848837/

Changelog 5.11.4:
- Full Changelog (104 Changes): https://lwn.net/Articles/848616/

But it will take a bit, im trying to compile the new kernel with AOCC(v3.0 Clang LLVM 12)+LTO(LLD)+O3 for Zen 2 & Zen 3
If someone wants to try AOCC, you need this patch: https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=c75173a26948363bdd11a0d5b90bd012ce4cc2e7

Wish me luck xD

@Ramalama2
Copy link

@fabianishere

While the complete building process of 5.11.7 runs really well, i get almost no warnings anymore. Seems like either the compiler or code was improved...

Im running on building 5.11.7 into this issue:

# finalize
/sbin/depmod -b debian/pve-edge-kernel-5.11.7-1/ 5.11.7-1
depmod: WARNING: could not open modules.order at /root/pve-edge-kernel/build/debian/pve-edge-kernel-5.11.7-1//lib/modules/5.11.7-1: No such file or directory
depmod: WARNING: could not open modules.builtin at /root/pve-edge-kernel/build/debian/pve-edge-kernel-5.11.7-1//lib/modules/5.11.7-1: No such file or directory
# Autogenerate blacklist for watchdog devices (see README)
install -m 0755 -d debian/pve-edge-kernel-5.11.7-1/lib/modprobe.d
ls debian/pve-edge-kernel-5.11.7-1/lib/modules/5.11.7-1/kernel/drivers/watchdog/ > watchdog-blacklist.tmp
ls: cannot access 'debian/pve-edge-kernel-5.11.7-1/lib/modules/5.11.7-1/kernel/drivers/watchdog/': No such file or directory
make[1]: *** [debian/rules:166: .install_mark] Error 2
make[1]: Leaving directory '/root/pve-edge-kernel/build'
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2
make: *** [Makefile:88: pve-edge-kernel-5.11.7-1_5.11.7-1_amd64.deb] Error 2

I guess there is something missing from ubuntu/debian to build the install packages?

@Ramalama2
Copy link

Screenshot 2021-03-19 012108

Yeah, okay, i gotcha...

I didn't knew that i need to checkout a specific ubuntu-mainline version xD

@Ramalama2
Copy link

Ramalama2 commented Mar 19, 2021

Changelog 5.11.7:

Removed from Kernel:

-d CONFIG_XEN \
-d CONFIG_DRM_RADEON \
-d CONFIG_DRM_AMDGPU \
-d CONFIG_DRM_NOUVEAU \
-d CONFIG_DRM_I915 \
-d CONFIG_DRM_XEN \
-d CONFIG_DRM_GMA500 \
-d CONFIG_SND_FIREWIRE \
-d CONFIG_SND_XEN_FRONTEND \
-d CONFIG_HYPERV \
-d CONFIG_SURFACE_PLATFORMS \
-d CONFIG_NET_VENDOR_GOOGLE \
-d CONFIG_CHROME_PLATFORMS \
-d CONFIG_MICROCODE_INTEL \
-d CONFIG_NFC \
-d CONFIG_MAC80211 \
-d CONFIG_CFG80211 \
-d CONFIG_BT \
-d CONFIG_RTL8192U \
-d CONFIG_RTLLIB \
-d CONFIG_WIMAX \
-d CONFIG_ANDROID \

Didn't managed to build AOCC with LTO yet, running into some errors that needs fixing first....

@Ramalama2
Copy link

For everyone that's interrested:
https://cloud.golima.de/s/E7KiWLFgzSp4MNP
Please keep in mind, to revert apparmor features back to stock.

I provide there Znver3 buils, build based on official pve 5.11 repo.
If someone wants to build his own kernel for another microarchitecture, use (kernel-update.sh) script.
There are no modules removed, it's 1:1 official, just updated and build with znver3.
There probably won't be 5.12 kernels.

Cheers

@fabianishere fabianishere added the wontfix This will not be worked on label Apr 19, 2021
@fabianishere
Copy link
Owner

Hi, I have decided not to offer pre-built kernels for non-generic microarchitectures anymore. The performance benefit of optimizing the kernel for a specific microarchitecture is unfortunately negligible and I don't have the time to maintain multiple builds.

@Ramalama2
Copy link

Ramalama2 commented Apr 25, 2021

i see it similar after all the time of doing this and testing.
But it took me long to come to this conclusion.
I will still provide some builds, but it's gonna happen 1-2x a month, not for every change anymore.

Thanks for your effort fabian, i learned here much!

Edit: Probably you can simply provide an "core2" build. There is a minimal performance benefit, and everyone can use it. Even an 2012 atom has sse3.
But that's up to you. Just can be the golden middle 😂

However, thank you very much and cheers 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request wontfix This will not be worked on
Projects
None yet
Development

No branches or pull requests

3 participants