[Allen & Heath Qu-Pac] regression since 0.3.53+ / no audio inputs unless manual selection of config profile ("Pro") in pulse audio control #16

ianhundere · 2022-07-09T05:12:46Z

since updating to 0.3.53 audio is no longer inputted to my daw, bitwig 4.3.1.

i confirmed this was the issue by using timeshift to revert before the update, running bitwig to confirm audio was being inputted and then updated to 0.3.53.

maybe we can rebase to 0.3.54 to see if that fixes it?

https://gitlab.freedesktop.org/pipewire/pipewire/-/blob/master/NEWS

ianhundere · 2022-07-11T14:49:52Z

i've since reverted / held back pipewire until 0.3.54 is available.

jacobgkau · 2022-07-11T17:41:08Z

I am not able to recreate this issue-- I'm able to record audio from a microphone into Bitwig using PipeWire 0.3.53. In the BitWig audio settings, do you have PipeWire selected for the Driver model option, or something else? Do you have an input bus added with the correct input selected? What computer and audio interface hardware are you using?

#17 will update to 0.3.54 once testing is complete.

ianhundere · 2022-07-11T20:03:49Z

i'll just reiterate that my setup is working with 0.3.52 but not 0.3.53, and by that i mean, none of my inputs work from my interface.

In the BitWig audio settings, do you have PipeWire selected for the Driver model option, or something else?
yes

Do you have an input bus added with the correct input selected?
yes...i have dozens of inputs and nothing has changed other than updating pipewire.

What computer and audio interface hardware are you using?
thinkpad t14s / allen heath qupac

again, i tested by reverting back to a working backup that had 0.3.52 installed with everything working (all 32 inputs working) and then updated to 0.3.53 to confirm.

all signs point to this being an issue caused by the 0.3.53 update.

lemme know if you need any further information from me.

ianhundere · 2022-07-14T13:12:26Z

@jacobgkau i noticed the PR was closed to update pipewire. when will that be available to download / test ?

mmstick · 2022-07-14T13:16:12Z

Today. Soon

jacobgkau · 2022-07-14T15:11:33Z

The update was released by this PR, please let us know if it fixes the issue.

ianhundere · 2022-07-14T19:49:53Z

awesome, thanks for this.

looks like it's working, but! i had to do the following for whatever reason:

open pulseaudio volume control
go to configuration
unselect the profile type (even though it was the correct profile; "Pro") for my interface
choose whatever else
then choose the "Pro" config profile again

this must get reset behind the scenes, or something of the sort. thanks again.

ianhundere · 2022-07-18T00:14:52Z

reopening as i'm having to do the above steps each and every time my interface resets.

marksumm · 2022-07-18T23:18:33Z

I'm seeing something similar with Behringer X18 mixers since I upgraded Pipewire. One or more devices (I have 2 mixers and an SPL Madicon MADI interface) won't pass audio until "Pro Audio" is explicitly deselected and re-selected. While the system is in this state, the transport in Bitwig hangs when the play button is pressed and then the audio engine disconnects. I'm also seeing segfaults from Pipewire in my syslog:

Jul 19 00:05:45 imhotep kernel: [ 2651.386255] pipewire[21971]: segfault at 0 ip 0000000000000000 sp 00007f2d697879e8 error 14 in pipewire[55e62c687000+1000]

ianhundere · 2022-07-19T02:24:42Z

@marksumm wanna post your issue here too to get more 👀 on it?

https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2567

ianhundere · 2022-07-19T14:40:13Z

@jacobgkau looks like the latest release of pipewire may fix a few of the regressions "...as well as some channel mapping regressions."

https://gitlab.freedesktop.org/pipewire/pipewire/-/releases/0.3.56

ianhundere · 2022-07-19T17:45:52Z

i wonder if it would cause issues if updates were pulled directly from the pw ppa. i'll try that and report back.
ex: sudo add-apt-repository ppa:pipewire-debian/pipewire-upstream

as per @mmstick, i'll not do that 😉

jacobgkau · 2022-07-19T19:27:36Z

@jacobgkau looks like the latest release of pipewire may fix a few of the regressions "...as well as some channel mapping regressions."

This PR will bring 0.3.56: #19

If you'd like to confirm whether this new release fixes the issue with these DACs with many input channels, you can now install it with these commands:

git clone https://github.com/pop-os/pop
./pop/scripts/apt add pipewire
sudo apt update
sudo apt upgrade

I would reboot after that to make sure you're running the new version. When you're finished testing, you can remove the test repository from your system with these commands:

./pop/scripts/apt remove pipewire
sudo apt update
sudo apt upgrade

If the new release found in that PR fixes this issue, then I'll link the issue to the PR.

ianhundere · 2022-07-19T19:37:15Z

@jacobgkau cheers, thanks so much for this. i'll test tonight / report back.

marksumm · 2022-07-19T19:59:06Z

@jacobgkau Thanks for that. Unfortunately, I still have both the profiles issue and the segfault issue. I've confirmed that the new version of pipewire is being used.

Unfortunately, the issues discussed in this thread are only the tip of a very big iceberg, which appeared during the last batch of updates (both OS and Bitwig). I don't want to pollute the thread and it's well past the point of being able to conveniently separate issues into bug reports directed at individual projects or products. In general, things seem much happier when there are no X18 mixers attached, so I'm going to expand my AD/DA and connect everything to that instead. This will remove the need to initialise and maintain synchronisation across multiple devices.

mmstick · 2022-07-19T20:45:36Z

Be sure to report issues to the developer's repository. This is purely a packaging fork so the best I can do is backport new releases from the development repository.

ianhundere · 2022-07-19T21:11:56Z

same issue here in that i have to unselect / reselect the "Pro" config profile in pulse audio control.

ianhundere · 2022-07-20T03:27:14Z

@mmstick added
pop-os/pop#2535

ianhundere · 2022-07-20T03:27:30Z

ps when will wireplumber be updated ?

marksumm · 2022-07-20T07:13:30Z

Just to let you know, this staging release of Pipewire seems to have stabilised my setup when using JACK mode in Bitwig 4.3.2 (which also contained some JACK-related fixes), although I had to create a new audio configuration and recreate my inputs and outputs first. So, either there are issues with Pipewire affecting only native clients, or Pipewire mode in Bitwig is broken.

Also created this issue in the Pipewire project: https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2577

marksumm · 2022-07-20T10:05:57Z

@ianhundere Have you tried Bitwig 4.3.2? Asking because I switched back to the .deb release this morning and now I'm no longer having profiles issues (still using the pipewire 0.3.56 staging release). See https://downloads.bitwig.com/4.3.2/Release-Notes-4.3.2.html

ianhundere · 2022-07-20T12:55:10Z

@ianhundere Have you tried Bitwig 4.3.2? Asking because I switched back to the .deb release this morning and now I'm no longer having profiles issues (still using the pipewire 0.3.56 staging release). See https://downloads.bitwig.com/4.3.2/Release-Notes-4.3.2.html

It's funny, I am using that but I noticed that it's no longer labeled as a pre-release candidate. I'll try installing it again and see. Thanks for the heads up.

edit: @marksumm just noticed your comment before your last about revertitback to JACK. is the profile issue fixed when using JACK ?

ianhundere · 2022-07-20T13:34:24Z

@marksumm okay, i just tested this and this is the behavior:

try to start bitwig...audio engine continually spins and i am forced to quit bitwig
try using JACK (since creating a new config for it) / no audio from inputs
^ i'm still forced to unselect / reselect the config profile in pulseaudio volume ctrl 😢

marksumm · 2022-07-20T13:47:25Z

@ianhundere Using 4.3.2 I have no profile issue with either JACK or Pipewire (at least so far today). I mentioned it because you said you were using 4.3.1 I still have other issues including the spinning on the first attempt to start.

By the way, you can terminate the Bitwig audio engine process without quitting the whole application by right-click near the transport position.

Update: I just encountered the profiles issue again. I'm now wondering if what really fixed it before was having api.acp.auto-profile = false set in my previous Wireplumber config.

ianhundere · 2022-07-20T14:35:51Z

@marksumm yah, i updated to 4.3.2 when it was a release candidate. i should have mentioned that.

hmm, weird...well i played around here a bit more and there were times the profile issues didn't happen. (using pipewire 0.3.56 and bitwig 4.3.2)
scenarios:

issue appears / i unselect / reselect profile / issue is gone
restart computer, but not interface / issue is gone
i turn off the interface and restart computer / issue is back
restart computer w/ interface on / issue is gone
restart interface / issue is gone

i'm really not sure what to make of it.

as for wireplumber, i mentioned that we were 2 versions behind here, pop-os/wireplumber#5, so hopefully that gets merged here soon for us to test.

edit: @marksumm i turned off my interface / restarted the computer and i didn't have to select a config profile. weird...i wanna say it's fixed, but i don't know.

@jacobgkau when 0.3.56 is officially released for pop, will we need to do anything special since we installed it via the repo ?

jacobgkau · 2022-07-20T14:53:16Z

@jacobgkau when 0.3.56 is officially released for pop, will we need to do anything special since we installed it via the repo ?

Just removing the testing repo like I explained in my comment:

./pop/scripts/apt remove pipewire
sudo apt update
sudo apt upgrade

You'll start getting 404 errors when running apt update after the branch is merged; you'll probably want to wait until the new version is released in https://github.com/pop-os/repo-release before running the apt upgrade. You may see an "update" since the commit hash might change during merging.

ianhundere · 2022-07-20T14:54:55Z

ah okay, so we should remove with the steps above and then update like normal when it's officially released.

ianhundere · 2022-07-20T15:25:25Z

@marksumm so after a bunch of restarts etc, i'm not experiencing either of the following:

bitwig doesn't stall anymore when starting up (ps thank you for the tip on terminating the audio engine process by right clicking the transport)
profile selection issue is gone

i'm hesitant to close this issue until 0.3.56 has been released officially.

jacobgkau · 2022-07-20T15:32:21Z

Linked the PipeWire PR to this issue since it sounds like the PR + a Bitwig update resolve the issue with these DACs.

marksumm · 2022-07-20T19:42:08Z

It does seem to be fixed. I think the one recurrence mentioned in my last update was in fact a total loss of synchronisation between audio devices. Probably re-selecting profiles caused them to be reinitialised.

ianhundere · 2022-07-31T03:02:32Z

@marksumm looks like this issue is back for me:
journalctl --no-hostname --user -u pipewire

Jul 30 14:39:43 systemd[1618]: Started PipeWire Multimedia Service.
Jul 30 14:39:43 pipewire[1635]: spa.alsa: hw:acp: Rate doesn't match (requested 44100, got 48000)
Jul 30 14:39:43 pipewire[1635]: spa.alsa: hw:acp: Rate doesn't match (requested 44100, got 48000)
Jul 30 14:39:43 pipewire[1635]: spa.alsa: hw:acp: Rate doesn't match (requested 44100, got 48000)
Jul 30 14:40:42 pipewire[1635]: spa.alsa: hw:3,0: snd_pcm_start: Broken pipe
Jul 30 14:40:42 pipewire[1635]: spa.audioadapter: 0x56374b3cf218: can't send command 2: Broken pipe
Jul 30 14:40:42 pipewire[1635]: pw.node: (alsa_input.usb-Allen_Heath_Ltd_QU-PAC-01.pro-input-0-110) suspended -> error (Start error: Broken>

then i unselect / reselect "Pro" via pulseaudio volume control:

Jul 30 22:54:35 pipewire[1632]: pw.link: 0x55618e69d5d0: one of the nodes is in error out:error in:suspended
Jul 30 22:54:35 pipewire[1632]: pw.link: 0x55618e69e200: one of the nodes is in error out:error in:suspended
Jul 30 22:54:35 pipewire[1632]: spa.alsa: hw:3,0: snd_pcm_start: Broken pipe
Jul 30 22:54:35 pipewire[1632]: spa.audioadapter: 0x55618e007e58: can't send command 2: Broken pipe
Jul 30 22:54:35 pipewire[1632]: spa.alsa: hw:3,0: snd_pcm_start: Broken pipe
Jul 30 22:54:35 pipewire[1632]: spa.audioadapter: 0x55618e007e58: can't send command 2: Broken pipe
Jul 30 22:54:35 pipewire[1632]: spa.alsa: hw:3,0: snd_pcm_start: Broken pipe
Jul 30 22:54:35 pipewire[1632]: spa.audioadapter: 0x55618e007e58: can't send command 2: Broken pipe
Jul 30 22:54:36 pipewire[1632]: spa.alsa: hw:3,0: snd_pcm_start: Broken pipe
Jul 30 22:54:36 pipewire[1632]: spa.audioadapter: 0x55618e007e58: can't send command 2: Broken pipe
Jul 30 22:54:36 pipewire[1632]: spa.alsa: hw:acp: Rate doesn't match (requested 44100, got 48000)
Jul 30 22:54:36 pipewire[1632]: spa.alsa: hw:3,0: snd_pcm_start: Broken pipe
Jul 30 22:54:36 pipewire[1632]: spa.audioadapter: 0x55618e007e58: can't send command 2: Broken pipe
Jul 30 22:54:36 pipewire[1632]: spa.alsa: hw:3,0: snd_pcm_start: Broken pipe
Jul 30 22:54:36 pipewire[1632]: spa.audioadapter: 0x55618e007e58: can't send command 2: Broken pipe
Jul 30 22:54:46 pipewire[1632]: spa.alsa: hw:3,0: follower delay:520 target:256 thr:256, resync

ianhundere · 2022-08-05T13:05:51Z

latest alsa update fixed this

ianhundere · 2022-10-05T15:22:26Z

@mmstick any way we could open this backup?

the issue popped back up a bit ago, i opened these as well that i'm tracking.

https://gitlab.freedesktop.org/pipewire/pipewire/-/issues/2567

https://bugzilla.kernel.org/show_bug.cgi?id=216519

mmstick · 2022-10-05T15:26:31Z

@ianhundere Would you happen to be using 0.3.59?

ianhundere · 2022-10-05T15:30:00Z

@ianhundere Would you happen to be using 0.3.59?

@mmstick 0.3.58 for me, has 0.3.59 been released for pop?

edit: any tips on testing/upgrading to 0.3.59?

mmstick · 2022-10-05T15:47:46Z

You can add it with sudo apt-manage add popdev:master. Should be released soon with the next round of updates.

ianhundere · 2022-10-05T15:50:47Z

@mmstick and final question, how do i go about reverting ?

thanks for the awesome support / quick responses.

mmstick · 2022-10-05T15:52:35Z

sudo apt-manage remove popdev-master

ianhundere · 2022-10-05T15:55:32Z

no go 😢

Oct 05 11:54:20 pipewire[1700]: spa.alsa: hw:5,0: snd_pcm_start: Broken pipe
Oct 05 11:54:20 pipewire[1700]: spa.audioadapter: 0x55860f861928: can't send command 2: Broken pipe
Oct 05 11:54:20 pipewire[1700]: pw.node: (alsa_input.usb-Allen_Heath_Ltd_QU-PAC-01.pro-input-0-112) suspended -> error (Start error: Broken pipe)
Oct 05 11:54:57 pipewire[1700]: pw.link: 0x55860fcd6930: one of the nodes is in error out:error in:running
Oct 05 11:54:57 pipewire[1700]: pw.link: 0x55860fcd7480: one of the nodes is in error out:error in:running
Oct 05 11:54:57 pipewire[1700]: pw.link: 0x55860fc6a8c0: one of the nodes is in error out:error in:running

ianhundere · 2022-10-18T17:00:37Z

@mmstick what's the suggested way of reverting to an older version of pipewire, 0.3.52, until this issue is resolved?

would i rebase myself?

mmstick · 2022-10-18T17:04:57Z

You'd have to manually build the debian packages from the commit you want to revert to, and install them all the same time.

git reset --hard {{commit_sha}}
dpkg-buildpackage -b
sudo dpkg -i ../*pipewire*.deb

ianhundere · 2022-10-18T17:13:24Z

okay, i'll try to revert to 0.3.52 / 31bf631

thanks for your support.

ianhundere · 2022-10-18T17:45:44Z

@mmstick

okay, so when i reverted obviously the debian directory no longer exists...so i pulled that out, reverted, and then put debian back and now receive:

❯ dpkg-buildpackage -b
dpkg-buildpackage: info: source package pipewire
dpkg-buildpackage: info: source version 0.3.59-0pop1
dpkg-buildpackage: info: source distribution jammy
dpkg-buildpackage: info: source changed by Michael Aaron Murphy <mmstick@pm.me>
dpkg-buildpackage: info: host architecture amd64
 dpkg-source --before-build .
dpkg-checkbuilddeps: error: Unmet build dependencies: doxygen libavahi-client-dev libbluetooth-dev libcanberra-dev libfdk-aac-dev libfreeaptx-dev libldacbt-abr-dev libldacbt-enc-dev liblilv-dev libpulse-dev libsbc-dev libsdl2-dev libsystemd-dev libusb-1.0-0-dev libv4l-dev libwebrtc-audio-processing-dev python3-docutils
dpkg-buildpackage: warning: build dependencies/conflicts unsatisfied; aborting
dpkg-buildpackage: warning: (Use -d flag to override.)

so i assume i need all those packages before i build.

ianhundere · 2022-10-26T23:40:06Z

@mmstick i got as far as:

Depends: libbluetooth3 (= 5.64-0ubuntu1) but 5.64-0ubuntu1pop0~1653326539~22.04~17ba3e9 is to be installed

i guess i could spin up an ubuntu container to build.

ianhundere · 2022-10-26T23:58:36Z

got a bit further in an ubuntu container but:

Running custom install script '/usr/bin/sh -c ln -fs pw-cat $DESTDIR/usr/bin/pw-play'
Running custom install script '/usr/bin/sh -c ln -fs pw-cat $DESTDIR/usr/bin/pw-record'
Running custom install script '/usr/bin/sh -c ln -fs pw-cat $DESTDIR/usr/bin/pw-midiplay'
Running custom install script '/usr/bin/sh -c ln -fs pw-cat $DESTDIR/usr/bin/pw-midirecord'
Running custom install script '/usr/bin/sh -c ln -fs pw-cat $DESTDIR/usr/bin/pw-dsdplay'
   dh_install
dh_install: warning: Cannot find (any matches for) "usr/lib/systemd/user/filter-chain.service" (tried in ., debian/tmp)

dh_install: warning: pipewire missing files: usr/lib/systemd/user/filter-chain.service
dh_install: warning: Cannot find (any matches for) "usr/share/pipewire/filter-chain.conf" (tried in ., debian/tmp)

dh_install: warning: pipewire-bin missing files: usr/share/pipewire/filter-chain.conf
dh_install: warning: Cannot find (any matches for) "usr/bin/pipewire-avb" (tried in ., debian/tmp)

dh_install: warning: pipewire-avb missing files: usr/bin/pipewire-avb
dh_install: warning: Cannot find (any matches for) "usr/share/pipewire/pipewire-avb.conf" (tried in ., debian/tmp)

dh_install: warning: pipewire-avb missing files: usr/share/pipewire/pipewire-avb.conf
dh_install: warning: Cannot find (any matches for) "usr/lib/*/spa-0.2/avb/libspa-avb.so" (tried in ., debian/tmp)

dh_install: warning: libspa-0.2-avb missing files: usr/lib/*/spa-0.2/avb/libspa-avb.so
dh_install: error: missing files, aborting
make: *** [debian/rules:7: binary] Error 255
dpkg-buildpackage: error: debian/rules binary subprocess returned exit status 2

It is not enough for `buffer` to be alive in its current scope because when execution enters that branch, `format` will be set to `fmt`, which points inside `buffer`. And since `format` is used outside that scope, `buffer` must live longer. This was detected by ASAN when Audacity was starting up. ==25007==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7ffdbcfef560 at pc 0x7fe44ca95db3 bp 0x7ffdbcfeeda0 sp 0x7ffdbcfeed90 READ of size 4 at 0x7ffdbcfef560 thread T0 #0 0x7fe44ca95db2 in spa_pod_parser_pod ../spa/include/spa/pod/parser.h:67 #1 0x7fe44ca9a805 in spa_format_parse ../spa/include/spa/param/format-utils.h:44 #2 0x7fe44cad293a in port_set_format ../spa/plugins/audioconvert/audioconvert.c:1934 #3 0x7fe44cadad14 in impl_node_port_set_param ../spa/plugins/audioconvert/audioconvert.c:2038 #4 0x7fe44ca587e2 in configure_format ../spa/plugins/audioconvert/audioadapter.c:509 #5 0x7fe44ca60dff in negotiate_format ../spa/plugins/audioconvert/audioadapter.c:822 #6 0x7fe44ca62bbf in impl_node_send_command ../spa/plugins/audioconvert/audioadapter.c:846 #7 0x7fe45ea1c2f1 in node_update_state ../src/pipewire/impl-node.c:407 #8 0x7fe45ea5137e in pw_impl_node_set_state ../src/pipewire/impl-node.c:2251 #9 0x7fe45eb3355f in pw_work_queue_destroy ../src/pipewire/work-queue.c:142 #10 0x7fe45b2cd6f4 in source_event_func ../spa/plugins/support/loop.c:615 #11 0x7fe45b2c634f in loop_iterate ../spa/plugins/support/loop.c:452 #12 0x7fe45e9ebebc in spa_hook_list_clean ../spa/include/spa/utils/hook.h:395 #13 0x5561e03dc722 in main ../src/daemon/pipewire.c:131 #14 0x7fe45da3c28f (/usr/lib/libc.so.6+0x2328f) #15 0x7fe45da3c349 in __libc_start_main (/usr/lib/libc.so.6+0x23349) #16 0x5561e03db2a4 in _start ../sysdeps/x86_64/start.S:115 Address 0x7ffdbcfef560 is located in stack of thread T0 at offset 160 in frame #0 0x7fe44ca56fa9 in configure_format ../spa/plugins/audioconvert/audioadapter.c:475 This frame has 4 object(s): [32, 36) 'state' (line 493) [48, 56) 'fmt' (line 494) [80, 128) 'b' (line 492) [160, 4256) 'buffer' (line 491) <== Memory access at offset 160 is inside this variable

ianhundere closed this as completed Jul 14, 2022

ianhundere reopened this Jul 18, 2022

jacobgkau changed the title ~~0.3.53 causing issues~~ [Allen & Heath Qu-Pac] 0.3.53+ causing issues Jul 18, 2022

ianhundere changed the title ~~[Allen & Heath Qu-Pac] 0.3.53+ causing issues~~ [Allen & Heath Qu-Pac] regression since 0.3.53+ causing issues Jul 18, 2022

ianhundere changed the title ~~[Allen & Heath Qu-Pac] regression since 0.3.53+ causing issues~~ [Allen & Heath Qu-Pac] regression since 0.3.53+ / no audio inputs unless manual selection of config profile ("Pro") in pulse audio control Jul 18, 2022

ianhundere mentioned this issue Jul 20, 2022

[Allen & Heath Qu-Pac] regression since 0.3.53+ / no audio inputs unless manual selection of config profile ("Pro") in pulse audio control pop-os/pop#2535

Closed

jacobgkau linked a pull request Jul 20, 2022 that will close this issue

0.3.56 Rebase #19

Merged

mmstick closed this as completed in #19 Jul 20, 2022

ianhundere mentioned this issue Sep 23, 2022

[Allen & Heath Qu-Pac] regression since 0.3.53+ / no audio inputs unless manual selection of config profile ("Pro") in pulse audio control alsa-project/alsa-lib#272

Closed

ianhundere mentioned this issue Nov 30, 2022

0.3.61 Rebase #33

Closed

[Allen & Heath Qu-Pac] regression since 0.3.53+ / no audio inputs unless manual selection of config profile ("Pro") in pulse audio control #16

[Allen & Heath Qu-Pac] regression since 0.3.53+ / no audio inputs unless manual selection of config profile ("Pro") in pulse audio control #16

Comments

ianhundere commented Jul 9, 2022 • edited Loading

ianhundere commented Jul 11, 2022

jacobgkau commented Jul 11, 2022

ianhundere commented Jul 11, 2022 • edited Loading

ianhundere commented Jul 14, 2022

mmstick commented Jul 14, 2022

jacobgkau commented Jul 14, 2022

ianhundere commented Jul 14, 2022

ianhundere commented Jul 18, 2022

marksumm commented Jul 18, 2022

ianhundere commented Jul 19, 2022

ianhundere commented Jul 19, 2022

ianhundere commented Jul 19, 2022 • edited Loading

jacobgkau commented Jul 19, 2022

ianhundere commented Jul 19, 2022

marksumm commented Jul 19, 2022

mmstick commented Jul 19, 2022

ianhundere commented Jul 19, 2022

ianhundere commented Jul 20, 2022

ianhundere commented Jul 20, 2022

marksumm commented Jul 20, 2022 • edited Loading

marksumm commented Jul 20, 2022 • edited Loading

ianhundere commented Jul 20, 2022 • edited Loading

ianhundere commented Jul 20, 2022 • edited Loading

marksumm commented Jul 20, 2022 • edited Loading

ianhundere commented Jul 20, 2022 • edited Loading

jacobgkau commented Jul 20, 2022

ianhundere commented Jul 20, 2022

ianhundere commented Jul 20, 2022

jacobgkau commented Jul 20, 2022

marksumm commented Jul 20, 2022

ianhundere commented Jul 31, 2022

ianhundere commented Aug 5, 2022

ianhundere commented Oct 5, 2022 • edited Loading

mmstick commented Oct 5, 2022

ianhundere commented Oct 5, 2022 • edited Loading

mmstick commented Oct 5, 2022 • edited Loading

ianhundere commented Oct 5, 2022

mmstick commented Oct 5, 2022

ianhundere commented Oct 5, 2022 • edited Loading

ianhundere commented Oct 18, 2022

mmstick commented Oct 18, 2022

ianhundere commented Oct 18, 2022 • edited Loading

ianhundere commented Oct 18, 2022 • edited Loading

ianhundere commented Oct 26, 2022

ianhundere commented Oct 26, 2022

ianhundere commented Jul 9, 2022 •

edited

Loading

ianhundere commented Jul 11, 2022 •

edited

Loading

ianhundere commented Jul 19, 2022 •

edited

Loading

marksumm commented Jul 20, 2022 •

edited

Loading

marksumm commented Jul 20, 2022 •

edited

Loading

ianhundere commented Jul 20, 2022 •

edited

Loading

ianhundere commented Jul 20, 2022 •

edited

Loading

marksumm commented Jul 20, 2022 •

edited

Loading

ianhundere commented Jul 20, 2022 •

edited

Loading

ianhundere commented Oct 5, 2022 •

edited

Loading

ianhundere commented Oct 5, 2022 •

edited

Loading

mmstick commented Oct 5, 2022 •

edited

Loading

ianhundere commented Oct 5, 2022 •

edited

Loading

ianhundere commented Oct 18, 2022 •

edited

Loading

ianhundere commented Oct 18, 2022 •

edited

Loading