systemd: 239 -> 242 #61321

andir · 2019-05-11T21:28:54Z

Motivation for this change

This is a follow-up to #56184.

As explained in #56184 (comment) I rebased @Mic92's attempt for systemd version 242 and added the two patches that I wrote last year.

The current head of my systemd fork is at https://github.com/andir/systemd/commits/nixos-v242. I had to add one additional patch to make udev rules work again (andir/systemd@7283141).

While running all the NixOS release.nix tests I discovered a few issues that required changes to our modules. Those are included in this PR.

From all the failing tests only the mysqlReplication test is seemingly relevant. It requires some refactoring in order to work on v242. The failure log can be seen here: https://gist.github.com/a46e172235ae9066afd758cbf9e86564 It would be great if someone with knowledge in the area could have a look. It was broken before. Nothing we've to take care of.

I am in the process of switching my notebook to this branch to see if anything that we aren't testing is obviously broken.

Feedback from others is highly appreciated.

My private hydra build of release.nix: https://hydra.h4ck.space/eval/155 (ipv6 only)

If we like these changes it would be nice if someone with access to https://github.com/nixos/systemd could push my systemd changes to the nixos-v242 branch.

Things done

andir · 2019-05-11T21:43:35Z

nixos/modules/services/hardware/udev.nix

@@ -282,6 +278,8 @@ in

    services.udev.path = [ pkgs.coreutils pkgs.gnused pkgs.gnugrep pkgs.utillinux udev ];

+    boot.kernelParams = mkIf (!config.networking.usePredictableInterfaceNames) [ "net.ifnames=0" ];


Does this work for all our cases? Previously we did exclude a udev rule. That could have affected containers and similar constructs. Now we only affect "real" systems.

tl;dr I don't think this should affect containers or custom network namespace applications.

As I understand it, the interface renaming should only effectively applied to physical interfaces (or if the kernel declares it as predictable) because the physical location is coded into the interface name (see NamePolicy in systemd.link(5) manpage). When creating virtual netdevs like vlans or bridges a custom name has to be supplied anyway. When moving physical interfaces to containers the interface has already been renamed on the host.

andir · 2019-05-11T21:46:30Z

nixos/modules/tasks/network-interfaces-systemd.nix

@@ -59,7 +59,14 @@ in
          in {
            DHCP = override (dhcpStr cfg.useDHCP);
          } // optionalAttrs (gateway != [ ]) {
-            gateway = override gateway;
+            routes = override [


I am not convinced this is the best way to solve this right now.

The issue was that we would add onlink routes for all network devices on the system. Introduced in systemd/systemd@4912ab7 systemd would set the onlink attribute for all routes that didn't come with static addresses.

It's certainly not ideal but then again adding the default gateway to every interface is FUBAR anyway. This fix is IMHO fine until we finally deprecate networking.defaultGateway and move it to the per interface configurations.

👍 I created an issue for this (#61629) and also added it to our systemd project board https://github.com/NixOS/nixpkgs/projects/22. (I am trying to populate that more with all the things we come across wile working on this PR).

pkgs/os-specific/linux/systemd/default.nix

ghost · 2019-05-12T09:58:25Z

I'm testing this now

andir · 2019-05-12T10:06:36Z

You might want to use `nixos-rebuild boot`. I lost control of my pid1 after switching. No idea what we can do about that.

…

On Sun, 12 May 2019, 11:58 Milan, ***@***.***> wrote: I'm testing this now — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#61321 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAE365F4R27JOIPPSMSPWTDPU7S4RANCNFSM4HMJAV2A> .

ghost · 2019-05-12T11:44:57Z

Yup, saw that too late, this is what it looked like:
https://termbin.com/scs1
After a reboot everything seems to work fine until now. I will report here if I find any issues.

flokli · 2019-05-12T12:38:22Z

we should probably also check why the failure with #56265 (comment) can't be reproduced anymore, and drop the systemd patch referenced there, if it's not effective anymore.

flokli · 2019-05-12T23:09:23Z

I figured out why we can't reproduce the error anymore - see explanation in #56265 (comment).

@andir, could you incorporate flokli@92600a9 into this PR, and drop andir/systemd@51077a9 from your systemd branch?

I'll try having a look at how to best fix nixops send-keys.

Mic92 · 2019-05-13T05:51:18Z

@andir loosing control as in no longer being able to contact via dbus?

ghost · 2019-05-14T09:49:35Z

Not 100% certain this is related
After switching to a new system configuration, systemd-timesyncd started crashing:

May 14 11:47:12 tachyon.pbb.lc systemd[21126]: systemd-timesyncd.service: Failed to set up special execution directory in /var/lib: Not a directory
May 14 11:47:12 tachyon.pbb.lc systemd[21126]: systemd-timesyncd.service: Failed at step STATE_DIRECTORY spawning /nix/store/5rnzb2cr3vw536px3p156d1g3arf18v0-systemd-242/lib/systemd/systemd-timesyncd: Not a directory

/var/lib exists and is a directory.

andir · 2019-05-14T13:41:32Z

@andir loosing control as in no longer being able to contact via dbus?

Yes, the output looked very similar to #61321 (comment).

Not 100% certain this is related
After switching to a new system configuration, systemd-timesyncd started crashing:

May 14 11:47:12 tachyon.pbb.lc systemd[21126]: systemd-timesyncd.service: Failed to set up special execution directory in /var/lib: Not a directory
May 14 11:47:12 tachyon.pbb.lc systemd[21126]: systemd-timesyncd.service: Failed at step STATE_DIRECTORY spawning /nix/store/5rnzb2cr3vw536px3p156d1g3arf18v0-systemd-242/lib/systemd/systemd-timesyncd: Not a directory

/var/lib exists and is a directory.

I am having the same. It seems like the daemon never came up after switching to the new systemd version. I have some idea what might cause this and will debug this tonight.

ghost · 2019-05-14T14:11:13Z

The timesyncd issue has been reported for Arch-based distros too:
systemd/systemd#12131

nixos-discourse · 2019-05-14T15:01:37Z

This pull request has been mentioned on Nix community. There might be relevant details there:

https://discourse.nixos.org/t/what-are-your-goals-for-19-09/2875/22

flokli · 2019-05-14T21:18:42Z

There's some more WIP patches over at https://github.com/flokli/nixpkgs/commits/systemd-v242-fk (not merged in yet here).

Planning to add some more tests for LVM and crypted volumes, plus testing of the send-keys functionality, to ensure it doesn't break.

andir · 2019-05-14T21:52:13Z

I spent a bit of time on the systemd-timesyncd issue. Removing the old symlink `/var/lib/systemd/timesync` fixed it for me locally. It pointed to `../private/systemd/timesync` (`/var/lib/private/systemd/timesync`) since the unit was previously running with `DyanmicUser=1`. The issue here seems to be that even though the permissions were okay timesyncd doesn't follow symlinks (anymore?) and thus fails to access the path that was symlinked. Upon the next start the directory (/var/lib/systemd/timesync) was created. That is definitly not a fix that I'd like to ship to users. Looking at the changes bit more lead me to the upstream commit 162e0b7 [0] where dynamic users was removed from systemd-timesyncd. Following that to the related PR 10117 [1] referrenced downstream issues from Debian and Fedora. The Debian issue seems to be about some confusion regarding "unknown users" from DBus. Following the Debian issue led to the bug 9503 [3] where a lengthy discussion and a commit that we've been shipping in v239 [4]. So I didn't chase that route further. Looking at how Fedora dealt with the issue brought me to their package spec [5] file. They check if `/var/lib/systemd/timesync` is a symlink and in that case they remove it and move the data from `/var/lib/private/systemd/timesync` to `/var/lib/systemd/timesync`. At this point I think it might be a safe way to do that in an automated fashion when switching to a new generation - before starting systemd-timesyncd. Any thoughts? /cc @flokli @petabyteboy [0] systemd/systemd@162e0b7 [1] systemd/systemd#10117 [3] systemd/systemd#9503 [4] NixOS/systemd@b179909 [5] https://src.fedoraproject.org/rpms/systemd/blob/b9896dc2cebc3c4cfda0a6c355183f645e372765/f/systemd.spec#_606

andir · 2019-05-15T07:13:38Z

I dropped the commit @flokli mentioned in #61321 (comment). Rerunning all the tests 🎉

nixos/doc/manual/release-notes/rl-1909.xml

edolstra · 2019-07-31T07:54:02Z

pkgs/os-specific/linux/systemd/default.nix

  # The interface version prevents NixOS from switching to an
  # incompatible systemd at runtime.  (Switching across reboots is
  # fine, of course.)  It should be increased whenever systemd changes
  # in a backwards-incompatible way.  If the interface version of two
  # systemd builds is the same, then we can switch between them at
  # runtime; otherwise we can't and we need to reboot.
-  passthru.interfaceVersion = 2;
+  passthru.interfaceVersion = 3;


What's the reason for bumping interfaceVersion? This is really undesirable because it requires every user to reboot.

After re-exec the userspace was no longer able to talk to the daemon.

See #61321 (comment) & #61321 (comment). Unfortunately the paste did expire :/

If we come up with a better way to handle these kinds of scenarios that would be great.

Rebooting every once in a while after your pid 1 did a major version change doesn't sound unreasonable to me.

Sounds like an upstream systemd bug that should be reported there. AFAIK systemd is supposed to be able to re-exec itself across version changes.

With local-fs.target part of sysinit.target (NixOS#61321), we don't need to add it explicitly to certain units anymore, and can change dependencies like they are in other distros (I picked from Google's official CentOS 7 image here). Like them, use StandardOutput=journal+console to pipe google-*.service output to the serial console as well.

Since NixOS#61321, local-fs.target is part of sysinit.target again, meaning units without DefaultDependencies=no will automatically depend on it, and the manual set dependencies can be dropped.

Ma27 · 2019-09-10T15:47:08Z

@fpletz @andir Not too experienced with udev and systemd internals, but this seems to break when using a predictably named network interface during a stage-1 boot (in my case to provide SSH access in initrd in order to enter the LUKS passphrase remotely). Until now I've used the 80-net-setup-links.rules from nixpkgs as described in NixOS discourse for a predictably named network interface which is configured like this:

{
  boot.kernelParams = [
    "ip=<v4-addr>::<gateway>:255.255.0.0::ens3:none"
  ];
}

May I ask what's the recommended way to implement such a feature in NixOS 19.09? AFAICT only udev is started in stage 1, so there's no predicatably named network interface then. I also checked the 80-net-setup-links.rules from our systemd fork (https://github.com/NixOS/systemd/blob/nixos-v243/rules/80-net-setup-link.rules) which doesn't rename the eth0 interface in my case (not sure why though, sorry!).

This is why I decided to continue using the old udev rules from nixpkgs (https://github.com/NixOS/nixpkgs/blob/release-19.03/nixos/modules/services/hardware/80-net-setup-link.rules) for now as my network is configured properly in stage-1 then, but I'm not sure if that's a suitable solution :)

Mic92 · 2019-09-11T10:49:28Z

I think this update might broke systemd --user for me. I am using lightdm as a login manager:

$ systemctl status --user
Failed to read server status: The name org.freedesktop.systemd1 was not provided by any .service files

andir · 2019-09-11T10:52:13Z

On Wed, 2019-09-11 at 03:49 -0700, Jörg Thalheim wrote: I think this update might broke systemd --user for me. I am using lightdm as a login manager.

What kind of error are you running into? My user session has been working like before. I am running a bunch of services (through home- manager) in my user session.

Mic92 · 2019-09-11T10:56:49Z

I still see my systemd user session, both the process as well as the dbus session in busctl - systemctl currently does not connect to those.

UPDATE need to have a look at that later. Currently I am at work.

andir · 2019-09-11T11:04:09Z

On Wed, 2019-09-11 at 03:56 -0700, Jörg Thalheim wrote: I still see my systemd user session, both the process as well as the dbus session - systemctl currently does not connect to those...

Ths is indeed odd. I can connect to mine just fine: ``` $ systemctl --user status timers.target ● timers.target - Timers Loaded: loaded (/nix/store/niw0jbw29x0rg85m7z8j5gll16y37g5n-systemd- 242/example/systemd/user/timers.target; static; vendor preset: enabled) Active: active since Tue 2019-09-10 00:04:14 CEST; 1 day 12h ago Docs: man:systemd.special(7) ``` Can you check what `strace -s 1500 -e connect,recvmsg,sendmsg -- systemctl --user status timers.target` produces?

Since NixOS#61321, local-fs.target is part of sysinit.target again, meaning units without DefaultDependencies=no will automatically depend on it, and the manual set dependencies can be dropped. (cherry picked from commit f74735c)

nixos-discourse · 2019-09-17T11:18:59Z

This pull request has been mentioned on NixOS Discourse. There might be relevant details there:

https://discourse.nixos.org/t/predictable-network-interface-names-in-initrd/4055/6

arianvp · 2019-09-17T14:14:50Z

I am having an issue with my systemd --user too, but slightly different. It is certainly running:

● user@0.service - User Manager for UID 0
   Loaded: loaded (/nix/store/niw0jbw29x0rg85m7z8j5gll16y37g5n-systemd-242/example/systemd/system/user@.service; static; vendor preset: enabled)
  Drop-In: /nix/store/kiiihixa3dcbb947sn4pwq5z90sksijn-system-units/user@.service.d
           └─overrides.conf
   Active: active (running) since Fri 2019-09-13 10:19:56 UTC; 4 days ago
     Docs: man:user@.service(5)
 Main PID: 25811 (systemd)
   Status: "Startup finished in 45ms."
       IP: 0B in, 0B out
    Tasks: 2
   Memory: 3.4M
      CPU: 113ms
   CGroup: /user.slice/user-0.slice/user@0.service
           └─init.scope
             ├─25811 /nix/store/4vw3gb6dk116y38vwgjv3ymq5mfdcfli-systemd-243/lib/systemd/systemd --user
             └─25812 (sd-pam)

Sep 13 10:19:56 arianvp.me systemd[1]: Starting User Manager for UID 0...
Sep 13 10:19:56 arianvp.me systemd[25811]: pam_unix(systemd-user:session): session opened for user root by (uid=0)
Sep 13 10:19:56 arianvp.me systemd[25811]: Reached target Paths.
Sep 13 10:19:56 arianvp.me systemd[25811]: Reached target Sockets.
Sep 13 10:19:56 arianvp.me systemd[25811]: Reached target Timers.
Sep 13 10:19:56 arianvp.me systemd[25811]: Reached target Basic System.
Sep 13 10:19:56 arianvp.me systemd[1]: Started User Manager for UID 0.
Sep 13 10:19:56 arianvp.me systemd[25811]: Reached target Main User Target.
Sep 13 10:19:56 arianvp.me systemd[25811]: Startup finished in 45ms.

but it cannot connect to the bus:

[root@arianvp:~]# systemctl status --user
Failed to connect to bus: No such file or directory

and if I look at systemctl status it seems there is no dbus.service running for the systemd --user instance:
strace:

(...snip..)
openat(AT_FDCWD, "/run/current-system/sw/lib/locale/locale-archive", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=128690032, ...}) = 0
mmap(NULL, 128690032, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f5331506000
close(3)                                = 0
openat(AT_FDCWD, "/proc/self/stat", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
read(3, "24430 (systemctl) R 24427 24427 "..., 1024) = 320
read(3, "", 1024)                       = 0
close(3)                                = 0
prlimit64(0, RLIMIT_NOFILE, {rlim_cur=512*1024, rlim_max=512*1024}, NULL) = 0
rt_sigaction(SIGBUS, {sa_handler=0x7f53397a9920, sa_mask=[], sa_flags=SA_RESTORER|SA_SIGINFO, sa_restorer=0x7f53396ac860}, {sa_handler=SIG_DFL, sa_mask=[], sa_flags=0}, 8) = 0
ioctl(1, TCGETS, {B38400 opost isig icanon echo ...}) = 0
newfstatat(AT_FDCWD, "/proc/1/root", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
newfstatat(AT_FDCWD, "/", {st_mode=S_IFDIR|0755, st_size=4096, ...}, 0) = 0
getpid()                                = 24430
socket(AF_UNIX, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 3
getsockopt(3, SOL_SOCKET, SO_RCVBUF, [212992], [4]) = 0
setsockopt(3, SOL_SOCKET, SO_RCVBUF, [8388608], 4) = 0
getsockopt(3, SOL_SOCKET, SO_SNDBUF, [212992], [4]) = 0
setsockopt(3, SOL_SOCKET, SO_SNDBUF, [8388608], 4) = 0
connect(3, {sa_family=AF_UNIX, sun_path="/run/user/0/systemd/private"}, 30) = -1 ECONNREFUSED (Connection refused)
(..snip...)
connect(3, {sa_family=AF_UNIX, sun_path="/run/user/0/bus"}, 18) = -1 ENOENT (No such file or directory)
close(3)                                = 0
openat(AT_FDCWD, "/nix/store/6yaj6n8l925xxfbcd65gzqx3dz7idrnn-glibc-2.27/share/locale/locale.alias", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0444, st_size=2997, ...}) = 0
read(3, "# Locale name alias data base.\n#"..., 4096) = 2997
read(3, "", 4096)                       = 0

writev(2, [{iov_base="Failed to connect to bus: No suc"..., iov_len=51}, {iov_base="\n", iov_len=1}], 2Failed to connect to bus: No such file or directory
) = 52
exit_group(1)                           = ?
+++ exited with 1 +++

Interestingly it also tries to connect to /run/user/0/systemd/private but fails. Whilst that file exists on my system

arianvp · 2019-09-19T10:14:12Z

FYI, rebooting my machine fixed this for me. It broke when doing nixos-rebuild switch between 239 -> 243 . We should probably tell people to reboot when going from 19.03 to 19.09

andir · 2019-09-19T10:38:50Z

If you were using the normal switching code you would have seen the incompatibility noted there.

…

On Thu, 19 Sep 2019, 12:14 Arian van Putten, ***@***.***> wrote: FYI, rebooting my machine fixed this for me. It broke when doing nixos-rebuild switch between 239 -> 243 . We should probably tell people to reboot when going from 19.03 to 19.09 — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#61321?email_source=notifications&email_token=AAE365DMJGYUYFUBS3IIEBLQKNGH7A5CNFSM4HMJAV2KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD7C6ZBI#issuecomment-533064837>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAE365D7E76KUVBOQHOOO7LQKNGH7ANCNFSM4HMJAV2A> .

edolstra · 2019-09-19T10:49:08Z

You're already required to reboot, but #68906 reverts that.

With local-fs.target part of sysinit.target (NixOS#61321), we don't need to add it explicitly to certain units anymore, and can change dependencies like they are in other distros (I picked from Google's official CentOS 7 image here). Like them, use StandardOutput=journal+console to pipe google-*.service output to the serial console as well. (cherry picked from commit 106a1fe)

flokli mentioned this pull request May 11, 2019

replace deprecated usage of PermissionsStartOnly (part 2) #56265

Merged

10 tasks

andir commented May 11, 2019

View reviewed changes

pkgs/os-specific/linux/systemd/default.nix Outdated Show resolved Hide resolved

ofborg bot added 6.topic: nixos 8.has: changelog 8.has: documentation 8.has: module (update) labels May 11, 2019

ofborg bot requested a review from edolstra May 11, 2019 22:12

ofborg bot added 10.rebuild-darwin: 101-500 10.rebuild-linux: 501+ 10.rebuild-linux: 5001+ labels May 11, 2019

andir requested review from fpletz and Mic92 May 12, 2019 09:11

andir force-pushed the nixos-v242 branch from 65541a7 to e5a5d08 Compare May 15, 2019 07:12

c0bw3b reviewed May 15, 2019

View reviewed changes

nixos/doc/manual/release-notes/rl-1909.xml Outdated Show resolved Hide resolved

c0bw3b reviewed May 15, 2019

View reviewed changes

nixos/doc/manual/release-notes/rl-1909.xml Outdated Show resolved Hide resolved

ofborg bot removed 6.topic: vim 2.status: merge conflict labels Jun 3, 2019

flokli merged commit 2812b5c into NixOS:staging Jun 3, 2019

flokli mentioned this pull request Jun 7, 2019

nixos/systemd: enable systemd-tmpfiles-setup and -clean for user sessions #62813

Merged

10 tasks

kliu128 mentioned this pull request Jun 19, 2019

Breakage due to systemd 242 parsing of \ in ExecStart= & other options #63533

Closed

flokli mentioned this pull request Jun 23, 2019

Containers with unified cgroup hierarchy are broken on 18.09 #48749

Closed

yorickvP mentioned this pull request Jul 16, 2019

systemd-timesyncd fails at step STATE_DIRECTORY on semi-clean install #64922

Closed

aristidb mentioned this pull request Jul 23, 2019

make nixos.stateVersion mandatory #65314

Open

8 tasks

edolstra reviewed Jul 31, 2019

View reviewed changes

flokli mentioned this pull request Aug 31, 2019

google-compute-config.nix: fix comments, update google-*.service units, fix paths in gce #67848

Merged

10 tasks

flokli mentioned this pull request Sep 1, 2019

nixos: remove dependencies on local-fs.target #67858

Merged

10 tasks

flokli mentioned this pull request Sep 1, 2019

nixos/redis: disable transparent huge pages (TLP) before starting Redis #67768

Merged

systemd: 239 -> 242 #61321

systemd: 239 -> 242 #61321

andir commented May 11, 2019 •

edited

andir May 11, 2019

fpletz May 17, 2019

andir May 11, 2019

fpletz May 17, 2019

andir May 17, 2019

ghost commented May 12, 2019

andir commented May 12, 2019 via email

ghost commented May 12, 2019

flokli commented May 12, 2019

flokli commented May 12, 2019

Mic92 commented May 13, 2019

ghost commented May 14, 2019 •

edited by ghost

andir commented May 14, 2019

ghost commented May 14, 2019

nixos-discourse commented May 14, 2019

flokli commented May 14, 2019

andir commented May 14, 2019 via email •

edited

andir commented May 15, 2019

edolstra Jul 31, 2019

andir Jul 31, 2019

edolstra Jul 31, 2019

Ma27 commented Sep 10, 2019

Mic92 commented Sep 11, 2019 •

edited

andir commented Sep 11, 2019 via email

Mic92 commented Sep 11, 2019 •

edited

andir commented Sep 11, 2019 via email

nixos-discourse commented Sep 17, 2019

arianvp commented Sep 17, 2019 •

edited

arianvp commented Sep 19, 2019

andir commented Sep 19, 2019 via email

edolstra commented Sep 19, 2019

		@@ -282,6 +278,8 @@ in

		services.udev.path = [ pkgs.coreutils pkgs.gnused pkgs.gnugrep pkgs.utillinux udev ];

		boot.kernelParams = mkIf (!config.networking.usePredictableInterfaceNames) [ "net.ifnames=0" ];

systemd: 239 -> 242 #61321

systemd: 239 -> 242 #61321

Conversation

andir commented May 11, 2019 • edited

Motivation for this change

Things done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ghost commented May 12, 2019

andir commented May 12, 2019 via email

ghost commented May 12, 2019

flokli commented May 12, 2019

flokli commented May 12, 2019

Mic92 commented May 13, 2019

ghost commented May 14, 2019 • edited by ghost

andir commented May 14, 2019

ghost commented May 14, 2019

nixos-discourse commented May 14, 2019

flokli commented May 14, 2019

andir commented May 14, 2019 via email • edited

andir commented May 15, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ma27 commented Sep 10, 2019

Mic92 commented Sep 11, 2019 • edited

andir commented Sep 11, 2019 via email

Mic92 commented Sep 11, 2019 • edited

andir commented Sep 11, 2019 via email

nixos-discourse commented Sep 17, 2019

arianvp commented Sep 17, 2019 • edited

arianvp commented Sep 19, 2019

andir commented Sep 19, 2019 via email

edolstra commented Sep 19, 2019

andir commented May 11, 2019 •

edited

ghost commented May 14, 2019 •

edited by ghost

andir commented May 14, 2019 via email •

edited

Mic92 commented Sep 11, 2019 •

edited

Mic92 commented Sep 11, 2019 •

edited

arianvp commented Sep 17, 2019 •

edited