Add prototype support for macvtap interfaces #2217

upxe · 2020-10-24T14:26:32Z

Firecracker uses /dev/net/tun to open/create tap interfaces. On the host these are typically bridged, using the standard Linux bridge implementation.

Linux macvtap interfaces are similar to bridges but simpler: the kernel doesn't snoop frames to maintain a FIB to map MAC addresses to interfaces and doesn't need to use flooding, because the full topology is known to the kernel.

However, macvtap interfaces are exposed to userland as /dev/tap%d device nodes, which Firecracker doesn't support. This commit adds support by checking whether the host_dev_name in the network configuration looks like a path name, and opening the device node if it does.

Tested thus:

    # Create a dummy interface to host the macvtap interfaces, plus two macvtap interfaces in bridge mode
    sudo ip link add dummy0 type dummy
    sudo ip link add link dummy0 name vtap0 type macvtap mode bridge
    sudo ip link add link dummy0 name vtap1 type macvtap mode bridge
    sudo ip link set dummy0 up
    sudo ip link set vtap0 up
    sudo ip link set vtap1 up
    sudo chmod 660 /dev/tap*             # For proof-of-concept purposes only
    sudo chown root:$(id -g) /dev/tap*   # For proof-of-concept purposes only

Configure two VMs thus; look in /dev for device node names; use ip link to get tap MAC address etc:

    {
        ...
            "network-interfaces" [
                {
                    ...
                    "host_dev_name: "/dev/tapXX",
                    "guest_mac": "<see host's ip link output>"
                }
            ]
        ...
    }

Start the VMs, give them IP addresses on the same subnet; they can then ping each other etc.

You must use the correct MAC address in the VM configuration, i.e. the MAC address that the host kernel has assigned to the tap interface.

Reason for This PR

macvtap (see #1933) allows VMs on a host to be networked without using the Linux bridge code; it is simpler and not prone to ARP spoofing.

The implementation in the commit proves that macvtap-based networking works; I'm after feedback please as to whether macvtap support would be considered for merging. If so then I'd look at configuring properly rather than leeching off the existing /dev/net/tun configuration code.

Alternatively, a more flexible approach would be to allow Firecracker to adopt a file descriptor as a network interface and read/write frames to it. (In other words, in this case, open /dev/tapXX as e.g. FD 3, then exec() firecracker and have it treat FD 3 as the tap interface). This would let me do other useful things on the host, but would require a jailer change to keep the relevant FD[s] open.

Description of Changes

Extend Tap::open_named() in virtio/net/tap.rs to support opening /dev/tapXX directly, rather than using /dev/net/tun to open/create tap interfaces. This allows Firecracker to use tap interfaces create via macvtap on the host.

This functionality can be added in rust-vmm.

License Acceptance

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

PR Checklist

I haven't looked at these as this is an RFC; if macvtap (or passing in FDs) is acceptable then I'll submit a PR that meets them.

[Reviewer TODO: Verify that these criteria are met. Request changes if not]

All commits in this PR are signed (git commit -s).
The reason for this PR is clearly provided (issue no. or explanation).
The description of changes is clear and encompassing.
Any required documentation changes (code and docs) are included in this PR.
Any newly added unsafe code is properly documented.
Any API changes are reflected in firecracker/swagger.yaml.
Any user-facing changes are mentioned in CHANGELOG.md.
All added/changed functionality is tested.

serban300 · 2020-10-27T09:09:52Z

Hi @upxe ,

Thanks for the contribution ! This might be very useful. We would like to do some performance tests first, and we can discuss next steps after. In the meanwhile, just dropping a few questions:

is this compatible with the snapshotting functionality ? For example right now when we restore from a snapshot we just recreate a tap with the same name as the old one. Do you see any complications here when using a macvtap instead ?
currently we have some unit tests that simulate frame tx/rx on the tap. Is this possible using the macvtap ?

upxe · 2020-10-27T20:25:40Z

is this compatible with the snapshotting functionality ? For example right now when we restore from a snapshot we just recreate a tap with the same name as the old one. Do you see any complications here when using a macvtap instead ?

I think that this depends on how you name your tap devices. The default (on Debian at any rate) isn't very useful: the XX in /tap/XX is the interface index, entirely unrelated to whatever name you gave the interface, and that appears in the ip link output. You might be able to convince udev (or whatever you use to populate /dev) to give you a better name, but just opening the same device node by name is probably a bad idea for these interfaces.

However, /sys/devices/virtual/net/<interface name>/macvtap/*/dev tells you what device node to open (or, indeed, to create in the jailer chroot). So: shouldn't be a problem to figure out what device to create or open, but you couldn't use the same code path as the existing tap device does.

This all means that we would probably need to distinguish between tap and macvtap interfaces at the configuration level, rather than relying on the interface name looking like /dev/something. That's probably a good idea anyway though.

currently we have some unit tests that simulate frame tx/rx on the tap. Is this possible using the macvtap ?

Haven't look into this yet - I raised the PR to see whether there was interest in principle in supporting this before diving in too deep. In theory, shouldn't be a problem, because once you've opened the device it's more-or-less a normal tap device. I shall have a look, but have had a few problems running the test suite - the whole unit test + integration test run (tools/devtool test) takes ages on my laptop, so I'm running cargo test within a shell in the Docker container. That seems to work, but a few tests fail in the current master.

gbionescu · 2020-11-09T07:37:36Z

Hi @upxe.
I just wanted to do a quick follow-up on this PR. Recently, I ran a quick test on this PR using iperf3 and noticed a slight decrease in throughput when reported to the master Firecracker version, but also saw that the overall CPU usage in the host is lower, which is something that we would like to look into more.

I will redo the tests and come back with the exact test scenario and numbers.

pclesr · 2020-11-18T12:53:47Z

Being able to specify the absolute path to the taped is extremely useful, as it allows unique names .

In an embedded environment, there is the desire to put each VM into a network namespace. Using MacVTap makes the configuration easier, as the Macvtap can be attached directly to the veth device in the network namespace without having to make another bridge. This simplifies the setup considerably. The environment is also sandboxed (like with the jailer) with other namespaces like mount, pid, etc to help isolate the environment.

acatangiu · 2020-11-25T09:52:57Z

Also asked for in #1933

gbionescu · 2020-11-26T16:58:24Z

Hi @upxe. I think that we can start working towards merging this, but the first step would be to point the PR towards a feature branch. I've created one called feature/macvtap - please update the PR when you have the time.

upxe · 2020-11-28T11:24:30Z

I've updated the PR; unless you're happy with the current approach (which was just a PoC, and I think is a bit too simple-minded), it would be useful to have a steer on how you want to proceed, e.g.

Whether you want to support both the current /dev/net/tun approach and macvtap interfaces, or see the macvtap approach replacing the existing one.
Whether checking for an absolute path is OK to use the macvtap implementation, or whether it look up the interface index given the interface name and see whether a suitable /dev/tapN exists. (Would probably play better with snapshotting.)

Apart from the above, I'm aware at the moment that I need to:

Add jailer support.
Ensure that snapshotting works. This will have to be aware of MAC addresses - the current code appears to persist the MAC address, in which case snapshot restoration will also have to set the host MAC address of the macvtap interface. I think that that's fine though.
Look into unit testing.
Consider doc changes.
Sign commits.

gbionescu · 2020-12-07T21:08:28Z

@upxe, indeed, all the items that you mentioned will have to be added, but I haven't looked into what the best approach would be to steer it towards the final implementation, given all the implications that it may have with snapshotting and other areas such as isolation between microVMs. This is why, as a first step, we should start working towards merging it into a feature branch.

Related to your first two bullets, preferably we would like to have both /dev/net/tun and macvtap available. Having absolute paths inside the snapshots would probably be a bad idea, correct.

Also, I would also test it through the Python framework, not only using rust unit tests. The snapshotting feature would also have to be tested through Python tests when macvtap snapshots are restored.

georgepisaltu · 2021-01-06T09:59:34Z

Hi @upxe!

Do you still plan on contributing to this PR?

upxe · 2021-01-06T19:02:38Z

Do you still plan on contributing to this PR?

I do; the day job intervened in December. Might get a chance to look at it this weekend.

btw, I had (thought that I'd) put it onto a feature branch on 28 Nov so was a little confused by @gc-plp's comment about "merging towards a feature branch" - if I haven't interpreted that correctly then please let me know.

Thanks.

gbionescu · 2021-01-06T19:13:00Z

@upxe, the PR is pointed to the correct branch now, but before merging we would need to have the items that you mentioned here, addressed.

casibbald · 2021-01-11T09:47:46Z

@upxe Been going through this and this PR is a great addition to FC, thank you so much. It happens to be functionality that is a blocker on a concept we are looking at.
Day job permitting, how are you progressing with the additional requirements to get the PR ready for final review and merging?

upxe · 2021-01-23T10:38:36Z

I'm looking at Jailer integration and there's a snag: macvtap interfaces are (AFAICT - shall investigate further) accessed via per-interface device nodes in /dev, rather than via /dev/net/tun. These individual device nodes aren't in the jailed firecracker's view of /dev, and the Jailer can't create then (like it does for /dev/kvm and /dev/net/tun) because it doesn't know which interfaces the VM will use.

Options that I can think of (unless I can somehow make the interfaces accessible via /dev/net/tun):

Put a read-only sysfs mount into the chroot (or bind-bound /sys/devices/virtual/net). Simplest and most flexible, but exposes rather more of the host to the VM than is ideal.
Jailer finds all macvtap interfaces (via sysfs) and creates device nodes in the chroot for all such interfaces that the firecracker uid/gid would be able to open. Potentially exposes unnecessary interfaces to the VM, but can be avoided by assigning a unique UID/GID to each VM, which is probably a good idea anyway. The macvtap interfaces must exist before running the Jailer.
Make a jailer user declare via command-line args the names of any macvtap interfaces that the VM will use, so that the Jailer can create the relevant device nodes. Puts minimal device nodes into the chroot, at the expense of some managerial/orchestration complexity.

Do the maintainers have any views on those? I'm minded to prefer option 2, unless the host may have lots of interfaces, in which case 1.

casibbald · 2021-01-23T13:23:41Z

Hi Upxe, could you drop me a private email? I have a few questions about the various implementation ideas. Kind regards Charles

…

On Sat, 23 Jan 2021 at 12:38, upxe ***@***.***> wrote: I'm looking at Jailer integration and there's a snag: macvtap interfaces are (AFAICT - shall investigate further) accessed via per-interface device nodes in /dev, rather than via /dev/net/tun. These individual device nodes aren't in the jailed firecracker's view of /dev, and the Jailer can't create then (like it does for /dev/kvm and /dev/net/tun) because it doesn't know which interfaces the VM will use. Options that I can think of (unless I can somehow make the interfaces accessible via /dev/net/tun): 1. Put a read-only sysfs mount into the chroot. Simplest and most flexible, but exposes rather more of the host to the VM than is ideal. 2. Jailer finds all macvtap interfaces (via sysfs) and creates device nodes in the chroot for all such interfaces that the firecracker uid/gid would be able to open. Potentially exposes unnecessary interfaces to the VM, but can be avoided by assigning a unique UID/GID to each VM, which is probably a good idea anyway. The macvtap interfaces must exist before running the Jailer. 3. Make a jailer user declare via command-line args the names of any macvtap interfaces that the VM will use, so that the Jailer can create the relevant device nodes. Puts minimal device nodes into the chroot, at the expense of some managerial/orchestration complexity. Do the maintainers have any views on those? I'm minded to prefer option 2. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#2217 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAA6C3Z4U6FI2CJXLM632FTS3KRLPANCNFSM4S5T274Q> .

gbionescu · 2021-01-25T13:43:42Z

@upxe I think that the question to ask here would be, what happens if a sandbox escape occurs and an attacker is able to access the jailer environment? Which of the three options would ensure that the attacker doesn't disturb a neighboring VM?

From this angle, option 3 seems to be ensure a minimal exposure of host devices inside the jailer. As you mentioned, there is some additional complexity passed on to the orchestrator, but a piece of extra logic would still have to be added in order to connect a microVM through a macvtap interface (e.g. creating the actual interface before launching the jailer).

What's your opinion here?

gbionescu · 2021-02-10T10:39:37Z

Hi @upxe
Have you had any chance to continue the work on this PR?

upxe · 2021-02-13T11:23:54Z

I'm happy to go with a command line argument in the jailer to have it create some additional device nodes.

The last remaining issue for now (haven't considered save/restore yet) is that I'm currently using sysfs to map interface names to device nodes, and sysfs isn't available in the jailed root. But if we're creating device nodes in the jailer then we just name them appropriately and have firecracker look for them if /sys isn't mounted.

If the host has a macvtap interface called foo that the user wants to use in a VM then we end up with:

User runs the jailer with e.g. --macvtap foo.
Jailer looks in host sysfs for the foo interface's major/minor, and creates /dev/net/foo in the jailed root.
When firecracker wants to open a network interface "foo":
1. If /dev/net/foo exists then open it and treat as a macvtap interface.
2. Else if /sys exists (i.e. no jailer) then see whether it's a macvtap interface, act as per jailer (and the existing code on this branch).
3. Else (no sys, no /dev/net/foo) use /dev/net/tun as per normal.

Incidentally, we can distinguish tap from mavctap via the ethtool GDRVINFO ioctl, but AFAICT we'd still need sysfs to figure out the major/minor numbers for the device node.

acatangiu · 2021-03-01T18:45:30Z

src/jailer/src/env.rs

+        // Since namespaces are shared by default when creating a new process using fork or clone,
+        // unshare() is used to disassociate the current process from the default shared namespace.
+        SyscallReturnCode(unsafe { libc::unshare(libc::CLONE_NEWNS) })


I would explicitly add in the comment that this unshares the mount namespace.

acatangiu · 2021-03-01T18:48:44Z

src/jailer/src/env.rs

+        .into_empty_result()
+        .map_err(Error::MountSysfs)?;
+
+        // Unmount the default sysfs.


nit:

Suggested change

// Unmount the default sysfs.

// Unmount the current sysfs since it's describing the previous namespace.

tests/integration_tests/functional/test_net.py

src/utils/src/net/macvtap.rs

src/devices/src/virtio/net/tap.rs

src/utils/src/net/macvtap.rs

Utilities to detect macvtap network interfaces. We're about to need them in both jailer and firecracker, so move from the latter into a separate module. Needs unit tests Signed-off-by: Ametros <github@ametros.net>

Firecracker network interfaces use /dev/net/tun to create tap interfaces. On the host these are typically bridged, using the standard Linux bridge implementation. Linux macvtap interfaces are similar to bridges but simpler: the kernel doesn't snoop frames to maintain a FIB to map MAC addresses to interfaces, because the full topology is known to the kernel. However, macvtap interfaces are exposed to userland as /dev/tap%d device nodes, which Firecracker doesn't support. This commit adds support by checking whether the `host_dev_name` looks like a path name, and opening the device node if it does. Tested thus: # Create a dummy interface to host the macvtap interfaces, plus two # macvtap interfaces in bridge mode sudo ip link add dummy0 type dummy sudo ip link add link dummy0 name vtap0 type macvtap mode bridge sudo ip link add link dummy0 name vtap1 type macvtap mode bridge sudo ip link set dummy0 up sudo ip link set vtap0 up sudo ip link set vtap1 up sudo chmod 666 /dev/tap* # For proof-of-concept purposes only # Configure two VMs thus; use ip link to get tap d { ... { "network-interfaces" [ { ... "host_dev_name: "/dev/tapX", "guest_mac": "<see ip link output>", } ] } ... } You must use the correct MAC address in the VM, i.e. the MAC address that the host kernel has assigned to the tap interface. To create a Tap instance given an interface name X, see whether /sys/devices/virtual/net/<X>/macvtap is a directory. If so then the interface is a macvtap interface, so look in that directory for a device name and open /dev/<that name>. Otherwise use the original open_named() code to open a tap interface via /dev/net/tun. Signed-off-by: Ametros <github@ametros.net>

macvtap interfaces are opened by device node, and (AFAICT) there's no way to discover the device major/minor numbers without sysfs, which isn't mounted in the jailer chroot. This commit adds a --macvtap=foo argument to the jailer; for each such argument, look up the device major/minor numbers for macvtap interface foo and create /dev/net/foo in the jailer chroot. Signed-off-by: Ametros <github@ametros.net>

Some other changes were made that make possible creating a set up for testing the macvtap support. Signed-off-by: Diana Popa <dpopa@amazon.com>

Signed-off-by: Diana Popa <dpopa@amazon.com>

This commit should be revised once merging to master since we want to add unit test for the functionality. Signed-off-by: Diana Popa <dpopa@amazon.com>

dianpopa

@gc-plp @georgepisaltu @acatangiu @luminitavoicu I addressed your feedback, PTAL! Thanks

src/utils/src/net/macvtap.rs

majek · 2022-09-02T14:07:52Z

Rebase please - the branch conflicts with main.

upxe · 2022-09-19T10:33:57Z

done on upxe:macvtap; don't know how to update this pull request because it's showing as merged + closed

dzenc · 2023-12-02T23:38:05Z

Despite the fact that this is marked merged + closed, it doesn't seem to be in there? Did this "die on the vine" somewhere? This seems like a lot of work, and a lot of value to be dropped. (I would like to make use of it!)

What can I do to help?

roypat · 2023-12-04T11:30:48Z

Hi @dzenc,

Despite the fact that this is marked merged + closed, it doesn't seem to be in there? Did this "die on the vine" somewhere? This seems like a lot of work, and a lot of value to be dropped. (I would like to make use of it!)

This PR only got merged to a feature branch, which never made it to main, which is why you cannot find it anywhere. We are tracking macVtap support in #1933, but have not continued working on it so far due to limited requests for this feature.

What can I do to help?

If you want, you can comment on #1933 with your usecase, to see if it can gain some traction again :)

dzenc · 2023-12-07T16:38:55Z

Comment added to #1933. Thanks for listening! :)

serban300 assigned upxe Oct 28, 2020

acatangiu mentioned this pull request Nov 20, 2020

Networking support is limited to using TUN/TAP host-side #158

Closed

4 tasks

acatangiu added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Nov 25, 2020

acatangiu requested a review from gbionescu November 25, 2020 09:54

upxe force-pushed the macvtap branch from 893e915 to c734a77 Compare November 28, 2020 10:51

upxe changed the base branch from master to feature/macvtap November 28, 2020 10:52

dianpopa added Status: Author and removed Status: Awaiting review Indicates that a pull request is ready to be reviewed labels Dec 15, 2020

upxe force-pushed the macvtap branch from c734a77 to e1681ba Compare January 9, 2021 13:20

upxe force-pushed the macvtap branch from e1681ba to 72f00d8 Compare February 13, 2021 13:46

raduiliescu linked an issue Feb 15, 2021 that may be closed by this pull request

[Network] Add support for MacVTap interfaces #1933

Open

dianpopa self-requested a review February 15, 2021 09:55

dianpopa added the Status: Awaiting review Indicates that a pull request is ready to be reviewed label Feb 16, 2021

dianpopa force-pushed the macvtap branch 2 times, most recently from e421ba8 to bb07771 Compare March 1, 2021 18:36

dianpopa requested a review from a team March 1, 2021 18:44

acatangiu reviewed Mar 1, 2021

View reviewed changes

gbionescu reviewed Mar 2, 2021

View reviewed changes

tests/integration_tests/functional/test_net.py Outdated Show resolved Hide resolved

dianpopa force-pushed the macvtap branch from bb07771 to 0de53c3 Compare March 2, 2021 14:01

georgepisaltu previously approved these changes Mar 2, 2021

View reviewed changes

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

src/devices/src/virtio/net/tap.rs Outdated Show resolved Hide resolved

luminitavoicu reviewed Mar 2, 2021

View reviewed changes

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

upxe and others added 8 commits March 2, 2021 18:09

macvtap: add utils/macvtap

1a31253

Utilities to detect macvtap network interfaces. We're about to need them in both jailer and firecracker, so move from the latter into a separate module. Needs unit tests Signed-off-by: Ametros <github@ametros.net>

CI: add macvtap parameter to the jailer

26cf606

Some other changes were made that make possible creating a set up for testing the macvtap support. Signed-off-by: Diana Popa <dpopa@amazon.com>

CI: class for a bridge containing macvtaps

52ab854

Signed-off-by: Diana Popa <dpopa@amazon.com>

CI: test for MacVTap functionality

fb28fb1

Signed-off-by: Diana Popa <dpopa@amazon.com>

Ammend license integration test

7f7c567

Signed-off-by: Diana Popa <dpopa@amazon.com>

Ammend coverages

c9c4be3

This commit should be revised once merging to master since we want to add unit test for the functionality. Signed-off-by: Diana Popa <dpopa@amazon.com>

dianpopa dismissed georgepisaltu’s stale review via c9c4be3 March 2, 2021 16:11

dianpopa force-pushed the macvtap branch from 0de53c3 to c9c4be3 Compare March 2, 2021 16:11

dianpopa reviewed Mar 2, 2021

View reviewed changes

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

src/utils/src/net/macvtap.rs Outdated Show resolved Hide resolved

dianpopa added Status: Awaiting review Indicates that a pull request is ready to be reviewed and removed Status: Author labels Mar 2, 2021

georgepisaltu approved these changes Mar 2, 2021

View reviewed changes

luminitavoicu approved these changes Mar 2, 2021

View reviewed changes

gbionescu merged commit 0a56fbf into firecracker-microvm:feature/macvtap Mar 3, 2021

dzenc mentioned this pull request Dec 7, 2023

[Network] Add support for MacVTap interfaces #1933

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add prototype support for macvtap interfaces #2217

Add prototype support for macvtap interfaces #2217

upxe commented Oct 24, 2020 •

edited by dianpopa

serban300 commented Oct 27, 2020

upxe commented Oct 27, 2020

gbionescu commented Nov 9, 2020

pclesr commented Nov 18, 2020

acatangiu commented Nov 25, 2020

gbionescu commented Nov 26, 2020

upxe commented Nov 28, 2020

gbionescu commented Dec 7, 2020

georgepisaltu commented Jan 6, 2021

upxe commented Jan 6, 2021

gbionescu commented Jan 6, 2021

casibbald commented Jan 11, 2021

upxe commented Jan 23, 2021 •

edited

casibbald commented Jan 23, 2021 via email

gbionescu commented Jan 25, 2021 •

edited

gbionescu commented Feb 10, 2021

upxe commented Feb 13, 2021

acatangiu Mar 1, 2021

dianpopa Mar 2, 2021

acatangiu Mar 1, 2021

dianpopa Mar 2, 2021

dianpopa left a comment

majek commented Sep 2, 2022

upxe commented Sep 19, 2022

dzenc commented Dec 2, 2023

roypat commented Dec 4, 2023

dzenc commented Dec 7, 2023

	// Unmount the default sysfs.
	// Unmount the current sysfs since it's describing the previous namespace.

Add prototype support for macvtap interfaces #2217

Add prototype support for macvtap interfaces #2217

Conversation

upxe commented Oct 24, 2020 • edited by dianpopa

Reason for This PR

Description of Changes

License Acceptance

PR Checklist

serban300 commented Oct 27, 2020

upxe commented Oct 27, 2020

gbionescu commented Nov 9, 2020

pclesr commented Nov 18, 2020

acatangiu commented Nov 25, 2020

gbionescu commented Nov 26, 2020

upxe commented Nov 28, 2020

gbionescu commented Dec 7, 2020

georgepisaltu commented Jan 6, 2021

upxe commented Jan 6, 2021

gbionescu commented Jan 6, 2021

casibbald commented Jan 11, 2021

upxe commented Jan 23, 2021 • edited

casibbald commented Jan 23, 2021 via email

gbionescu commented Jan 25, 2021 • edited

gbionescu commented Feb 10, 2021

upxe commented Feb 13, 2021

acatangiu Mar 1, 2021

Choose a reason for hiding this comment

dianpopa Mar 2, 2021

Choose a reason for hiding this comment

acatangiu Mar 1, 2021

Choose a reason for hiding this comment

dianpopa Mar 2, 2021

Choose a reason for hiding this comment

dianpopa left a comment

Choose a reason for hiding this comment

majek commented Sep 2, 2022

upxe commented Sep 19, 2022

dzenc commented Dec 2, 2023

roypat commented Dec 4, 2023

dzenc commented Dec 7, 2023

upxe commented Oct 24, 2020 •

edited by dianpopa

upxe commented Jan 23, 2021 •

edited

gbionescu commented Jan 25, 2021 •

edited