Missing kernel headers #4

alexreinert · 2020-05-28T12:16:17Z

The image contains the kernel package in version 1.20200527-1
But there is no matching header package available in the APT repository.

satmandu · 2020-05-28T15:29:52Z

For what it is worth I have been making functional arm64 header packages for the current firmware rpi kernels for use in ubuntu 64 bit on my RPI4 with this hacky script: https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

XECDesign · 2020-05-28T18:21:23Z

There's no good way to get a headers package without throwing the current kernel package out and doing it the right way from the start. That is a WIP, but won't ship in the near future.

StuartIanNaylor · 2020-05-29T09:01:39Z

There's no good way to get a headers package without throwing the current kernel package out and doing it the right way from the start. That is a WIP, but won't ship in the near future.

I was just trying a custom driver with DKMS and it progressed through armv7 correctly and just stopped on aarch64 because one didn't exist.

It looked like the addition raspberrypi-kernel-headers:aarch64 would work as expected with raspberrypi-kernel-headers:armhf but is just missing?

tomerrr · 2020-05-29T13:53:04Z

For what it is worth I have been making functional arm64 header packages for the current firmware rpi kernels for use in ubuntu 64 bit on my RPI4 with this hacky script: https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

i've just tried your script on Raspberry Pi OS which builds deb headers & kernel packages fine ,
but I'm unable to install them - when trying with sudo dpkg -i

error setting ownership of './boot/config-5.4.42-v8+': Operation not permitted
when trying to install the headers deb
and
error setting ownership of '/boot/dtbs.dpkg-new': Operation not permitted
on the linux kernel deb

any idea how to solve this?

XECDesign · 2020-05-29T13:55:46Z

It looked like the addition raspberrypi-kernel-headers:aarch64 would work as expected with raspberrypi-kernel-headers:armhf but is just missing?

Sorry, but no, it's not that simple. It's certainly possible but would add a lot of extra manual steps to the build process involving running things in containers of different architectures in two passes. That will waste me many hours in the future. That time would be better spent upfront fixing the package so that everything just builds as expected in the first place.

pik33 · 2020-05-29T18:23:19Z

@tomerr: you cannot set ownership/permission on /boot because it is fat32

Maybe unmounting /boot while performing the process and make the false /boot directory will help to finish the process.

tomerrr · 2020-05-29T18:50:55Z

@pik33 thank's for the tip! that's exactly what i did (unmounted /boot) and installed the headers deb. then it was just one file to copy to the real /boot directory after remounting it.
managed to install Wireguard that way.
thank's @satmandu for your excellent script

satmandu · 2020-06-01T01:01:35Z

@tomerrr @pik33 Sorry, I wrote that script for ubuntu, which has the fat32 partition mounted at /boot/firmware
I should probably look at the headers package on rasbian and see how the scripts are built there...

fatez · 2020-06-02T10:51:06Z

For what it is worth I have been making functional arm64 header packages for the current firmware rpi kernels for use in ubuntu 64 bit on my RPI4 with this hacky script: https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

Hi,

I am testing your script on a 4GB raspberry 4 with the following operating system : Raspberry Pi OS (64 bit) beta test version :

$ uname -ar
Linux enterprise 5.4.42-v8+ #1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64 GNU/Linux

I need linux-headers so I can install wireguard, but when I launch your script comes out with this error:

exit 0
dpkg-deb: error: failed to open package info file 'image//DEBIAN/control' for reading: No such file or directory
mv: cannot stat '/image.deb': No such file or directory

could you tell me what to do?

tomerrr · 2020-06-02T10:56:57Z

@fatez

$ uname -ar
Linux enterprise 5.4.42-v8+ #1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64 GNU/Linux

I need linux-headers so I can install wireguard, but when I launch your script comes out with this error:

exit 0
dpkg-deb: error: failed to open package info file 'image//DEBIAN/control' for reading: No such file or directory
mv: cannot stat '/image.deb': No such file or directory

could you tell me what to do?

i think you're not using the official raspberry kernel - uname should read

Linux raspberrypi 5.4.42-v8+ #1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64 GNU/Linux

fatez · 2020-06-02T11:02:52Z

@fatez

$ uname -ar
Linux enterprise 5.4.42-v8+ #1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64 GNU/Linux

I need linux-headers so I can install wireguard, but when I launch your script comes out with this error:

exit 0
dpkg-deb: error: failed to open package info file 'image//DEBIAN/control' for reading: No such file or directory
mv: cannot stat '/image.deb': No such file or directory

could you tell me what to do?

i think you're not using the official raspberry kernel - uname should read

Linux raspberrypi 5.4.42-v8+ #1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64 GNU/Linux

but "raspberry" or "enterprise" are the hostname

tomerrr · 2020-06-02T11:06:02Z

oh , sorry. i can run the script again to test what results am i getting now.
@fatez - script still work fine on Raspberry Pi OS beta . Would you like me to send you the created deb files?

fatez · 2020-06-02T11:49:35Z

you'd be very kind, but as soon as there's a kerel update, I'll be in the same situation.

I need to figure out what's wrong.

EDIT:

I understood the problem, I was doing it like this: sudo sh./make_deb.sh
and it didn't work, but once I did that: sudo chmod 777 make_deb.sh and run with : ./make_deb.sh worked!

satmandu · 2020-06-02T14:48:44Z

I'm about to load up the test 64-bit image... so I'll try to get the script fixed then.

satmandu · 2020-06-02T22:51:46Z

I had to keep my script from dropping the kernel config file into /boot and trying to set permissions on it.

Once that was commented out, the headers package installs fine.

0x73-w1337 · 2020-06-09T13:16:20Z

For what it is worth I have been making functional arm64 header packages for the current firmware rpi kernels for use in ubuntu 64 bit on my RPI4 with this hacky script: https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

The script works flawlessly and I've got kernel headers for 5.4.44-v8+ working without any issue. +1 Kudos!

pprindeville · 2020-07-09T22:41:58Z

There's no good way to get a headers package without throwing the current kernel package out and doing it the right way from the start. That is a WIP, but won't ship in the near future.

@XECDesign What qualifies as the "near future"? Are we past that yet?

XECDesign · 2020-07-10T00:25:27Z

A headers package may or may not show up at some point. I'd like to promise that it will, but something more important always comes along.

pprindeville · 2020-07-13T14:13:05Z

A headers package may or may not show up at some point. I'd like to promise that it will, but something more important always comes along.

Could I try to help out with that? What are the blocking issues?

esbeeb · 2020-07-15T16:52:03Z

Wireguard is a must for me, and it's very easy to install in Ubuntu 20.04 64bit for the RPi4. So that's why this bug means to a lot to me, to be resolved. I'd like for Wireguard to be easy to install, and not break every time some kernel security update is released, downloaded, and installed.

esbeeb · 2020-07-20T04:02:27Z

It looked like the addition raspberrypi-kernel-headers:aarch64 would work as expected with raspberrypi-kernel-headers:armhf but is just missing?

Sorry, but no, it's not that simple. It's certainly possible but would add a lot of extra manual steps to the build process involving running things in containers of different architectures in two passes. That will waste me many hours in the future. That time would be better spent upfront fixing the package so that everything just builds as expected in the first place.

Can you please tell us more about what a proper fix would entail, @XECDesign ? Can you please break it down for us?

leshniak · 2020-07-26T00:45:40Z

Wireguard is a must for me, and it's very easy to install in Ubuntu 20.04 64bit for the RPi4. So that's why this bug means to a lot to me, to be resolved. I'd like for Wireguard to be easy to install, and not break every time some kernel security update is released, downloaded, and installed.

Just build your own kernel if this is urgent. It's pretty straightforward. I've been doing it since a long time on the RPi itself. This is enough for extracting the headers (based on kernel scripts):

export KERNEL_VERSION=`strings arch/arm64/boot/Image | grep 'Linux version' | cut -d' ' -f3`
export KERNEL_HEADERS_PATH="../linux-headers-$KERNEL_VERSION"
export RSYNC_OPTS="-aAXv"

mkdir -pv $KERNEL_HEADERS_PATH
find . -name 'Makefile*' -o -name 'Kconfig*' -o -name '*.pl' | rsync $RSYNC_OPTS --files-from=- . $KERNEL_HEADERS_PATH
find arch/*/include include scripts -type f -o -type l | rsync $RSYNC_OPTS --files-from=- . $KERNEL_HEADERS_PATH
find arch/arm64 -name module.lds -o -name Kbuild.platforms -o -name Platform | rsync $RSYNC_OPTS --files-from=- . $KERNEL_HEADERS_PATH
find $(find arch/arm64 -name include -o -name scripts -type d) -type f | rsync $RSYNC_OPTS --files-from=- . $KERNEL_HEADERS_PATH
find tools/objtool -type f -executable | rsync $RSYNC_OPTS --files-from=- . $KERNEL_HEADERS_PATH
find arch/arm64/include Module.symvers include scripts -type f | rsync $RSYNC_OPTS --files-from=- . $KERNEL_HEADERS_PATH
find scripts/gcc-plugins -name '*.so' -o -name gcc-common.h | rsync $RSYNC_OPTS --files-from=- . $KERNEL_HEADERS_PATH
cp .config $KERNEL_HEADERS_PATH

Then move it to the /usr/src and re-link /lib/modules/$KERNEL_VERSION/build to that directory.

I'm using wireguard too – on arm64:

Linux raspberrypi 5.4.51-v8~f0eeafb #1 SMP PREEMPT Sun Jul 26 00:33:34 CEST 2020 aarch64 GNU/Linux
[ 41.456440] wireguard: WireGuard 1.0.20200712 loaded. See www.wireguard.com for information.

cfilipem · 2020-07-26T14:33:52Z

Now that we have a 64bit Raspberry Pi OS, the raspberrypi-kernel-headers package for arm64 would be something nice to have. It would allow easier kernel updates when we are running out-of-tree kernel modules.
@XECDesign, can you to increase the priority of fixing this?

T-Birth · 2020-08-02T12:47:49Z

For everyone landing on this Page.
I used the following Guide -> https://github.com/notasausage/pi-hole-unbound-wireguard
and had many troubles installing Wireguard on Pi OS64.

With this solution it finally worked:

WATCH OUT ! ->
Don't use something like "sudo rpi-update" with the current OS64 Beta if u want to get Wireguard working. Had so many troubles with this command.

Don't install raspberrypi-kernel-headers (its mentioned in the Guide above in the category "Setting Up a VPN with WireGuard". Just use this command instead ->
sudo apt install libelf-dev libmnl-dev build-essential git

Installed Pi OS64 from scratch

Used my little update command -> sudo apt update && sudo apt-get -y upgrade && sudo apt-get -y dist-upgrade && sudo apt-get -y autoremove && sudo apt-get -y autoclean

Used this script to build the kernel headers -> https://gist.github.com/satmandu/a507c59d84737f6d29ff353395819d51

Installed the builded image with -> sudo dpkg -i --force-all linux-image-5.4.51-v8+_arm64.deb (some error messages appeard but wayne)

Installed the builded headers with -> sudo dpkg -i --force-all linux-headers-5.4.51-v8+_arm64.deb

If dependencies are missing use -> sudo apt --fix-broken install

Now Proceed with the Guide above and the installation of Wireguard should work, even on 64.

T-Birth · 2020-08-02T14:48:19Z

copy pasted this nice script into this

nano make_deb.sh
chmod +x make_deb.sh
./make_deb.sh

sjjgo · 2021-01-01T14:39:54Z

Removing the non-arm64 headers and modules per @MichaIng's instructions did the job for me too. Presumably this is a fragile solution though – it will break with the next kernel update when new headers and modules are installed?

MichaIng · 2021-01-01T14:49:41Z

Good to know that it was not only us where it is required. Yes, unless there is something changed to fix non-arm64 WireGuard builds, the next package upgrade would fail the same way. Although you could make it permanent by adding a script to /etc/kernel/postinst.d/ that must be alphabetically ordered before the WireGuard DKMS builds script, to remove all non-arm64 kernels and/or headers.

Another cleaner solution would be to skip installing non-arm64 kernel modules + headers from the DEB packages in the first place via /etc/dpkg/dpkg.cfg.d/ exclusions. E.g.:

cat << '_EOF_' > /etc/dpkg/dpkg.cfg.d/skip_non_arm64_kernel
path-exclude /lib/modules/*+/*
path-include /lib/modules/*-v8+/*
path-exclude /usr/src/linux-headers-*+/*
path-include /usr/src/linux-headers-*v8+/*
_EOF_

But that does not work for rpi-update of course.

XECDesign · 2021-01-05T10:44:29Z

Added a 5.10 kernel to the 'untested' component with non-64bit files removed, if anyone would like to test it.

Wireguard dkms.conf from debian-backports doesn't try to build it for 5.10, but v4l2loopback-dkms seems to work as a test.

http://archive.raspberrypi.org/debian/pool/untested/r/raspberrypi-firmware/raspberrypi-kernel_1.20210105.master-1_arm64.deb

http://archive.raspberrypi.org/debian/pool/untested/r/raspberrypi-firmware/raspberrypi-kernel-headers_1.20210105.master-1_arm64.deb

http://archive.raspberrypi.org/debian/pool/untested/r/raspberrypi-firmware/raspberrypi-bootloader_1.20210105.master-1_arm64.deb

MichaIng · 2021-01-05T13:51:03Z

Great, many thanks! The current firmware (libraspberrypi0/-bin) libraries and binaries do work with this kernel build, don't they?

XECDesign · 2021-01-05T13:54:11Z

No promises

MichaIng · 2021-01-05T14:10:48Z

Fair enough. If anyone faces issues, affected binaries/libraries (/opt/vc/bin + /opt/vc/lib) can be manually updated from: https://github.com/raspberrypi/firmware/tree/master/hardfp/opt/vc
Only a small number have changed since current stable package release, so it's not a big deal.

tomerrr · 2021-01-08T20:08:24Z

Added a 5.10 kernel to the 'untested' component with non-64bit files removed, if anyone would like to test it.

Wireguard dkms.conf from debian-backports doesn't try to build it for 5.10, but v4l2loopback-dkms seems to work as a test.

http://archive.raspberrypi.org/debian/pool/untested/r/raspberrypi-firmware/raspberrypi-kernel_1.20210105.master-1_arm64.deb

http://archive.raspberrypi.org/debian/pool/untested/r/raspberrypi-firmware/raspberrypi-kernel-headers_1.20210105.master-1_arm64.deb

http://archive.raspberrypi.org/debian/pool/untested/r/raspberrypi-firmware/raspberrypi-bootloader_1.20210105.master-1_arm64.deb

looking good, Wireguard works for me


pi@pi4:~ $ dpkg-query -l raspberrypi-kernel*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                       Version             Architecture Description
+++-==========================-===================-============-================
ii  raspberrypi-kernel         1.20210108.master-1 arm64        Raspberry Pi boo
ii  raspberrypi-kernel-headers 1.20210108.master-1 arm64        Header files for
pi@pi4:~ $ uname -a
Linux pi4 5.10.5-v8+ #1391 SMP PREEMPT Thu Jan 7 17:55:54 GMT 2021 aarch64 GNU/Linux
pi@pi4:~ $ lsmod | grep wireguard
wireguard              73728  0
libchacha20poly1305    16384  1 wireguard
ip6_udp_tunnel         16384  1 wireguard
udp_tunnel             28672  1 wireguard
libcurve25519_generic    40960  1 wireguard
libblake2s             16384  1 wireguard
ipv6                  528384  60 wireguard

MichaIng · 2021-01-08T20:22:21Z

WireGuard is built into this kernel anyway, so headers and dkms are not required to get it running with this kernel, simply do apt install wireguard-tools and you're good to go 😉.

geerlingguy · 2021-02-08T23:13:56Z

It looks like headers weren't built for the latest kernel?

$ dpkg-query -l raspberrypi-kernel*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name               Version      Architecture Description
+++-==================-============-============-=================================
ii  raspberrypi-kernel 1.20210201-1 arm64        Raspberry Pi bootloader

MichaIng · 2021-02-08T23:21:42Z

It looks like you haven't installed them:

# apt show raspberrypi-kernel-headers
Package: raspberrypi-kernel-headers
Version: 1.20210201-1
Priority: optional
Section: kernel
Source: raspberrypi-firmware
Maintainer: Serge Schneider <serge@raspberrypi.org>
Installed-Size: 180 MB
Provides: linux-headers
Homepage: https://github.com/raspberrypi/firmware
Download-Size: 27.6 MB
APT-Sources: https://archive.raspberrypi.org/debian buster/main armhf Packages
Description: Header files for the Raspberry Pi Linux kernel
 This package provides the architecture-specific kernel header files for
 the Raspberry Pi Linux kernel, generally used for building out-of-tree
 kernel modules.

Joulinar · 2021-02-08T23:28:10Z

@MichaIng

but it is the 32bit apt package. isn't it?

APT-Sources: https://archive.raspberrypi.org/debian buster/main armhf Packages

MichaIng · 2021-02-08T23:41:23Z

This is automatically built for all architectures, including x86_64 and i386, maybe for cross-compiling: http://archive.raspberrypi.org/debian/pool/main/r/raspberrypi-firmware/raspberrypi-kernel-headers_1.20210201-1_arm64.deb
I also checked the package list, it's present and contains the correct updated file: http://archive.raspberrypi.org/debian/dists/buster/main/binary-arm64/Packages

esbeeb · 2021-03-16T20:27:13Z

So can the issue be closed then?

MichaIng · 2021-03-16T20:52:48Z

The only case I remember where a build failed, despite the kernel headers package, was with WireGuard which is shipped as builtin since Linux v5.10, so I'd say yes. Any build issues make more sense in an own issue.

XECDesign · 2021-03-16T22:21:10Z

Cheers, thanks for testing.

Pittconnect · 2021-06-12T23:40:45Z

Ok the header problem is back !! its now Kernel 5.4.79-V7 upgrade to 5.4.79-V8+

And Rebooting does not fix the problem

MichaIng · 2021-06-12T23:54:48Z

You're on the wrong repository here. You run an ARMv7 kernel, hence it's impossible that you use the Raspberry Pi OS 64-bit image.

What tools is it that shows you this wrong and misleading message? "v8+" wouldn't be a "new" kernel version, but the kernel for the ARMv8 architecture, hence for the 64-bit image indeed. Since your kernel is the ARMv7 one, that should stay as it is.

But it is correct that your kernel is outdated (current is 5.10.x), and furthermore you don't have the headers package installed, hence the missing headers. Do the following to upgrade it and install headers:

apt update
apt install raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader

Pittconnect · 2021-06-13T02:41:55Z

Thanks I did run the update and install and I have a question it INSTALLS vga666 is that for REAL ? [image: image.png] Thanks in advance Ra

…

On Sat, Jun 12, 2021 at 7:55 PM MichaIng ***@***.***> wrote: You're on the wrong repository here. You run an ARMv7 kernel, hence it's impossible that you use the Raspberry Pi OS *64-bit* image. What tools is it that shows you this wrong and misleading message? "v8+" wouldn't be a "new" kernel version, but the kernel for the ARMv8 architecture, hence for the 64-bit image indeed. Since your kernel is the ARMv7 one, that should stay as it is. But it is correct that your kernel is outdated, and furthermore you don't have the headers package installed, hence the missing headers. Do the following to upgrade it and install headers: apt update apt install raspberrypi-kernel raspberrypi-kernel-headers raspberrypi-bootloader — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#4 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHISA7BIKKI7PTWVSLK4PQDTSPXVJANCNFSM4NNBBQHQ> .

lurch · 2021-06-13T08:51:55Z

it INSTALLS vga666 is that for REAL ?

Yup https://github.com/raspberrypi/firmware/blob/master/boot/overlays/README#L3310 - the "666" just means 6 bits for red, 6 bits for green and 6 bits for blue.

HinTak · 2021-06-14T19:00:05Z

FWIW, I rather think that it is not 6-bit red / 6-bit green / 6 bit blue , but just 6 red + 6 green + 6 blue . 6 x 6 x 6 = 216, which is smaller than 255. This is a way of doing 8-bit colors using a 6 x 6 x 6 color cube. It knows about 216 different colors in total. Having vga at 8-bit color, using a 6x6x6 representation of colors, makes a lot of sense. It is a color vga with very "blocky" (non-24-bit) colors. Sometimes you uses vga at 16 bit color at a 5 bit red / 6 bit green / 5 bits blue representation . 5 + 6 + 5 = 16; or just waste one bit at 5-bit / 5-bit / 5-bit. Some would allocate one more bit to green which the eyes are more sensitive to. These are all very old representations, before 24-bit true colours on computers became the norm.

pelwell · 2021-06-14T19:37:44Z

FWIW, you're wrong - this uses 20 pins to provide 18-bit colour with 2 sync signals:

				brcm,pins = <2 3 4 5 6 7 8 9 10 11 12
					     13 14 15 16 17 18 19 20 21>;

HinTak · 2021-06-14T19:59:58Z

Oops. How's that 18-bit byte-aligned? That's more than 2-bytes and less than 3 - seems wasteful. One might want to do 24-bit color (padding and dropping the lower 2 bits) anyway, if 3-byte is being moved about...

pelwell · 2021-06-14T20:22:00Z

I believe the firmware will do the best it can with any source display format, e.g. a full 24-bit RGB, 565 etc. The overlay is just asking the kernel to set or preserve the alternate function selectors on those pins.

lurch · 2021-06-14T20:34:24Z

There's more info here: https://www.raspberrypi.org/documentation/hardware/raspberrypi/dpi/README.md

HinTak · 2021-06-14T22:02:42Z

Argh, thanks both of you for the raspberrypi display info! That's quite different from x86 land!

6by9 · 2021-06-15T09:21:20Z

RGB565 and RGB888 (and others) exist as input pixel formats to be rendered to a display device.

However the vga666 overlay is for a hardware device called the vga666 that converts the DPI interface (which I don't believe any x86 platform has ever supported) into VGA using a simple 6-bit per component resistor ladder. It's never a format that is stored in memory, so byte alignment is irrelevant.

HinTak · 2021-06-15T14:22:27Z

So vga666 is a HAT? That's cool. Fen Logic sounds like a company based in north east Cambridge...

6by9 · 2021-06-15T14:30:20Z

So vga666 is a HAT

Not a HAT as it doesn't have the ID EEPROM, but an add-on board, yes. It can be bought from the likes of Pi Supply, and various eBay sellers.

Fen Logic sounds like a company based in north east Cambridge...

The board was designed by Gert van Loo who is one of our former colleagues at Broadcom in Cambridge, and he open sourced that design via his side-line consultancy firm - https://www.fenlogic.com/index.html

tomerrr mentioned this issue Jun 11, 2020

unable to upgrade package linux-image-4.19.0-9 #54

Closed

pelwell mentioned this issue Jun 25, 2020

v4l2loopback #65

Open

pelwell mentioned this issue Jul 3, 2020

Kernel headers for the latest Raspberry OS 5.4.50-v8+ arm64 is missing raspberrypi/linux#3705

Closed

lurch mentioned this issue Jul 9, 2020

raspberrypi-kernel-headers mismatch with raspberrypi-kernel ... can't build out-of-tree kernel modules #71

Closed

pelwell mentioned this issue Jul 31, 2020

Not able to install wireguard #80

Open

XECDesign closed this as completed Mar 16, 2021

jgartrel mentioned this issue Mar 13, 2024

Add support for 64-bit kernels RPi-Distro/rpi-source#29

Merged

Missing kernel headers #4

Missing kernel headers #4

Comments

alexreinert commented May 28, 2020

satmandu commented May 28, 2020

XECDesign commented May 28, 2020

StuartIanNaylor commented May 29, 2020 • edited Loading

tomerrr commented May 29, 2020

XECDesign commented May 29, 2020

pik33 commented May 29, 2020 • edited Loading

tomerrr commented May 29, 2020 • edited Loading

satmandu commented Jun 1, 2020

fatez commented Jun 2, 2020 • edited Loading

tomerrr commented Jun 2, 2020

fatez commented Jun 2, 2020

tomerrr commented Jun 2, 2020 • edited Loading

fatez commented Jun 2, 2020 • edited Loading

satmandu commented Jun 2, 2020

satmandu commented Jun 2, 2020

0x73-w1337 commented Jun 9, 2020

pprindeville commented Jul 9, 2020

XECDesign commented Jul 10, 2020

pprindeville commented Jul 13, 2020

esbeeb commented Jul 15, 2020

esbeeb commented Jul 20, 2020

leshniak commented Jul 26, 2020 • edited Loading

cfilipem commented Jul 26, 2020

T-Birth commented Aug 2, 2020

T-Birth commented Aug 2, 2020

sjjgo commented Jan 1, 2021

MichaIng commented Jan 1, 2021 • edited Loading

XECDesign commented Jan 5, 2021 • edited Loading

MichaIng commented Jan 5, 2021

XECDesign commented Jan 5, 2021

MichaIng commented Jan 5, 2021

tomerrr commented Jan 8, 2021 • edited Loading

MichaIng commented Jan 8, 2021

geerlingguy commented Feb 8, 2021

MichaIng commented Feb 8, 2021

Joulinar commented Feb 8, 2021

MichaIng commented Feb 8, 2021

esbeeb commented Mar 16, 2021

MichaIng commented Mar 16, 2021 • edited Loading

XECDesign commented Mar 16, 2021

Pittconnect commented Jun 12, 2021

MichaIng commented Jun 12, 2021 • edited Loading

Pittconnect commented Jun 13, 2021 via email

lurch commented Jun 13, 2021

HinTak commented Jun 14, 2021 via email

pelwell commented Jun 14, 2021

HinTak commented Jun 14, 2021 via email

pelwell commented Jun 14, 2021

lurch commented Jun 14, 2021

HinTak commented Jun 14, 2021

6by9 commented Jun 15, 2021

HinTak commented Jun 15, 2021

6by9 commented Jun 15, 2021

StuartIanNaylor commented May 29, 2020 •

edited

Loading

pik33 commented May 29, 2020 •

edited

Loading

tomerrr commented May 29, 2020 •

edited

Loading

fatez commented Jun 2, 2020 •

edited

Loading

tomerrr commented Jun 2, 2020 •

edited

Loading

fatez commented Jun 2, 2020 •

edited

Loading

leshniak commented Jul 26, 2020 •

edited

Loading

MichaIng commented Jan 1, 2021 •

edited

Loading

XECDesign commented Jan 5, 2021 •

edited

Loading

tomerrr commented Jan 8, 2021 •

edited

Loading

MichaIng commented Mar 16, 2021 •

edited

Loading

MichaIng commented Jun 12, 2021 •

edited

Loading