Add Kernel Adress Sanitization support by vladandrew · Pull Request #191 · unikraft/unikraft

vladandrew · 2021-04-25T16:48:11Z

Kernel Address Sanitizer for Unikraft. Currently working only under KVM x86_64. We use the first part of the heap for the shadow memory(e.g. KASAN_MD_SHADOW_SIZE bytes are used for the shadow memory). We instrument the memory allocator to mark memory as either valid or invalid. We use the shadow memory to keep track of our marks.

Kasan instruments all memory accesses such that they are valid, if an memory region marked as invalid is accessed then we call UK_CRASH~. uk_malloc_ifpagesis instrumented such that the allocated memory is marked as valid and can be accessed,kasan_mark_invalid` marks the memory as invalid and causes the kernel to crash with a helpful message. We add a global KASAN flag and disable KASAN for several internal libraries such as the allocator.

Example of KASAN crash output when we do an out of bounds access of an array:

[    0.105388] CRIT: [libkasan] <kasan.c @  179> ===========KernelAddressSanitizer===========
[    0.107294] CRIT: [libkasan] <kasan.c @  179> ERROR:
[    0.108431] CRIT: [libkasan] <kasan.c @  179> * invalid access to address 0x25b01b
[    0.110177] CRIT: [libkasan] <kasan.c @  179> * write of size 1
[    0.111537] CRIT: [libkasan] <kasan.c @  179> * redzone code 0x2 (partial redzone)
[    0.113275] CRIT: [libkasan] <kasan.c @  179> ============================================

danield20

Looks good.

nderjung

Hi @vladandrew, can you please rebase?

@danield20 once Vlad has rebased, can you approve with your signature,

Thanks! 🐒

nderjung

Hey Vlad! Thanks for introducing this new library, it's pretty cool!

For this PR's archive and for future documentation, could you please add a paragraph or two on how KASAN works and how it will work in Unikraft.

Apart from the notes inline (mainly styling suggestions for better readability), I'm also receiving some compile-time warnings during its compilation:

warning: conflicting types for built-in function '__asan_load1_noabort'; expected
'void(void *)' [-Wbuiltin-declaration-mismatch]
  244 | void __asan_load##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:266:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  266 | DEFINE_ASAN_LOAD_STORE(1);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:248:6: warning: conflicting types for built-in function '__asan_store1_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]
  248 | void __asan_store##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:266:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  266 | DEFINE_ASAN_LOAD_STORE(1);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:244:6: warning: conflicting types for built-in function '__asan_load2_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]
  244 | void __asan_load##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:267:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  267 | DEFINE_ASAN_LOAD_STORE(2);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:248:6: warning: conflicting types for built-in function '__asan_store2_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]
  248 | void __asan_store##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:267:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  267 | DEFINE_ASAN_LOAD_STORE(2);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:244:6: warning: conflicting types for built-in function '__asan_load4_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]
  244 | void __asan_load##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:268:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  268 | DEFINE_ASAN_LOAD_STORE(4);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:248:6: warning: conflicting types for built-in function '__asan_store4_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]
  248 | void __asan_store##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:268:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  268 | DEFINE_ASAN_LOAD_STORE(4);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:244:6: warning: conflicting types for built-in function '__asan_load8_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]
  244 | void __asan_load##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:269:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  269 | DEFINE_ASAN_LOAD_STORE(8);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:248:6: warning: conflicting types for built-in function '__asan_store8_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]
  248 | void __asan_store##size##_noabort(uintptr_t addr)   \
      |      ^~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:269:1: note: in expansion of macro 'DEFINE_ASAN_LOAD_STORE'
  269 | DEFINE_ASAN_LOAD_STORE(8);
      | ^~~~~~~~~~~~~~~~~~~~~~
/usr/src/unikraft/unikraft/lib/kasan/kasan.c:244:6: warning: conflicting types for built-in function '__asan_load16_noabort'; expected 'void(void *)' [-Wbuiltin-declaration-mismatch]

vladandrew · 2021-12-01T18:41:07Z

Hi Alex,

Thank you for the review, I have solved all your comments. Let me know if there is anything else.

nderjung · 2021-12-02T12:04:29Z

Ok I was able to compile the application against redis.

It's the default settings plus kasan. However, running it bare-bones it is caught and crashes:

qemu-guest -k ./build/redis-x86_65-kvm -a "-- --help"
[...]
[    0.101175] CRIT: [libkasan] <kasan.c @  179> ===========KernelAddressSanitizer===========
[    0.105073] CRIT: [libkasan] <kasan.c @  179> ERROR:
[    0.107362] CRIT: [libkasan] <kasan.c @  179> * invalid access to address 0x4af010
[    0.111065] CRIT: [libkasan] <kasan.c @  179> * read of size 8
[    0.113788] CRIT: [libkasan] <kasan.c @  179> * redzone code 0x2 (partial redzone)
[    0.117335] CRIT: [libkasan] <kasan.c @  179> ============================================

Is this now expected behaviour? Before it would look like this (without kasan):

qemu-guest -k ./build/redis-x86_65-kvm -a "-- --help"
[...]
Powered by
o.   .o       _ _               __ _
Oo   Oo  ___ (_) | __ __  __ _ ' _) :_
oO   oO ' _ `| | |/ /  _)' _` | |_|  _)
oOo oOO| | | | |   (| | | (_) |  _) :_
 OoOoO ._, ._:_:_,\_._,  .__,_:_, \___)
                   Tethys 0.5.0~807598b
Usage: ./redis-server [/path/to/redis.conf] [options]
       ./redis-server - (read config from stdin)
       ./redis-server -v or --version
       ./redis-server -h or --help
       ./redis-server --test-memory <megabytes>

Examples:
       ./redis-server (run the server with default conf)
       ./redis-server /etc/redis/6379.conf
       ./redis-server --port 7777
       ./redis-server --port 7777 --replicaof 127.0.0.1 8888
       ./redis-server /etc/myredis.conf --loglevel verbose

Sentinel mode:
       ./redis-server /etc/sentinel.conf --sentinel
[    0.135880] CRIT: [libvfscore] <dentry.c @  216> Assertion failure: dp->d_refcnt > 0

When i re-compile with ukcpio and kasan and attach a .cpio file, I am now receiving:

qemu-guest -k ./build/redis_kvm-x86_64 -m 512 -i ./redis.cpio -a "-- /redis.conf"
[...]
Booting from ROM...
[    0.000000] Info: [libkvmplat] <setup.c @  269> Entering from KVM (x86)...
[    0.000000] Info: [libkvmplat] <setup.c @  270>      multiboot: 0x9500
[    0.000000] Info: [libkvmplat] <setup.c @  281>         initrd: 0x3a1000
[    0.000000] Info: [libkvmplat] <setup.c @  283>     heap start: 0x3a0000
[    0.000000] Info: [libkvmplat] <setup.c @  286>  heap start (2): 0x3b1000
[    0.000000] Info: [libkvmplat] <setup.c @  288>      stack top: 0x1ffd0000
[    0.000000] Info: [libkvmplat] <setup.c @  306> Switch from bootstrap stack to stack @0x1ffe0000
[    0.000000] Info: [libukboot] <boot.c @  202> Unikraft constructor table at 0x304000 - 0x304050
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x10a2f0())...
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x10b0d0())...
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x10db70())...
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x126690())...
[    0.000000] Info: [libuklibparam] <param.c @  113> libname: netdev, 72
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x12fbb0())...
[    0.000000] dbg:  [libvfscore] <main.c @ 1875> (int) uk_syscall_r_dup2((int) 0x0, (int) 0x1)
[    0.000000] dbg:  [libvfscore] <main.c @ 1822> (int) uk_syscall_r_dup3((int) 0x0, (int) 0x1, (int) 0x0)
[    0.000000] dbg:  [libvfscore] <main.c @ 1875> (int) uk_syscall_r_dup2((int) 0x0, (int) 0x2)
[    0.000000] dbg:  [libvfscore] <main.c @ 1822> (int) uk_syscall_r_dup3((int) 0x0, (int) 0x2, (int) 0x0)
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x13b1e0())...
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x143f40())...
[    0.000000] Info: [libuklibparam] <param.c @  113> libname: vfs, 96
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x108620())...
[    0.000000] dbg:  [libukbus] <bus.c @   52> Register bus handler: 0x344000
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x109030())...
[    0.000000] dbg:  [libukbus] <bus.c @   52> Register bus handler: 0x344060
[    0.000000] dbg:  [libukboot] <boot.c @  206> Call constructor: 0x1149f0())...
[    0.000000] dbg:  [libuklibparam] <param.c @  598> Library argument ends at 0
[    0.000000] Info: [libukboot] <boot.c @  216> Found 1 library args
[    0.000000] Info: [libukboot] <boot.c @  224> Initialize memory allocator...
[    0.000000] dbg:  [libukboot] <boot.c @  231> Try memory region: 0x3a0000 - 0x3a1000 (flags: 0x31)...
[    0.000000] CRIT: [libukallocbbuddy] <bbuddy.c @  502> Assertion failure: max > min
[    0.000000] Info: [libkvmplat] <shutdown.c @   35> Unikraft halted

When i disable kasan but keep ukcpio, it works fine.

vladandrew · 2021-12-02T12:31:19Z

Hi Alex,

In the first case, the behaviour seems right since KASAN detects an invalid access which may be related to the assertion failing in the default case. [ 0.135880] CRIT: [libvfscore] <dentry.c @ 216> Assertion failure: dp->d_refcnt > 0

In the second case, I think that you need to allocate more memory to the VM because KASAN uses part of the heap for the shadow memory.

PS: We have tested this implementation with Nginx and Redis for the FlexOS paper.

When the host indicates support for TSO, enable it and accept large packets when the corresponding uknetbuf flag is set. Signed-off-by: Marco Schlumpp <marco@unikraft.io> Reviewed-by: Eduard Vintilă <eduard.vintila47@gmail.com> Reviewed-by: Andrei Tatar <andrei@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GitHub-Closes: unikraft#1088

In the current form they impose very strong synchronization requirements, making it impossible to implement a more efficient read-write lock design. Most rwlocks sidestep this by either using a weaker `try_upgrade` variant or having a separate `lock_upgradable` function (allowing multiple readers but only one upgradable lock/writer lock). This removes these two functions for now to prevent new code relying on these functions. This change is breaking because it already was part of the previous release. Signed-off-by: Marco Schlumpp <marco@unikraft.io> Reviewed-by: Andrei Tatar <andrei@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GitHub-Closes: unikraft#1090

Although `checkpatch.pl` checks for commit message lines to not be longer than `75` characters, in the case of `Github`'s UI, a commit subject longer than `70` character results in its line overflowing into the next. E.g. ``` [Selector]/[Component]: Lorem ipsum dolor sit amet, consectetuer adipiscin ``` becomes ``` [Selector]/[Component]: Lorem ipsum dolor sit amet, consectetuer adip... ...iscin ``` Since this is not aesthetically pleasing, enforce a character limit for commit message subjects to not be longer than `70` characters. Signed-off-by: Sergiu Moga <sergiu@unikraft.io> Reviewed-by: Stefan Jumarea <stefanjumarea02@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GitHub-Closes: unikraft#1077

This code was imported and this occurrence of the type was not adjusted when the type name changed. Signed-off-by: Marco Schlumpp <marco@unikraft.io> Reviewed-by: Eduard Vintilă <eduard.vintila47@gmail.com> Reviewed-by: Rares Miculescu <miculescur@gmail.com> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GiHub-Closes: unikraft#1076

This attribute allows to opt-out of the standard C strict aliasing rule. Signed-off-by: Marco Schlumpp <marco@unikraft.io> Reviewed-by: Eduard Vintilă <eduard.vintila47@gmail.com> Reviewed-by: Rares Miculescu <miculescur@gmail.com> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GiHub-Closes: unikraft#1076

The macro violates the strict aliasing rules. By using the newly introduced may_alias attribute we can prevent miscompilations because of that. Signed-off-by: Marco Schlumpp <marco@unikraft.io> Reviewed-by: Eduard Vintilă <eduard.vintila47@gmail.com> Reviewed-by: Rares Miculescu <miculescur@gmail.com> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GiHub-Closes: unikraft#1076

Since `start16` related symbols have a relocation mechanism similar, but entirely independent from `ukreloc`, rename all such symbols, macro's and their uses so that they do not contain the `uk_reloc` or the `UKRELOC` substrings. Signed-off-by: Sergiu Moga <sergiu@unikraft.io> Reviewed-by: Razvan Virtan <virtanrazvan@gmail.com> Reviewed-by: Marco Schlumpp <marco@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GitHub-Closes: unikraft#1113

Allow having the same relocation mechanism of the SIPI vector symbols be used regardless of whether `ukreloc` or `PIE` are enabled or not. With this commit, SMP initialization has now the same SIPI behavior regardless of `ukreloc`/`PIE`, thus dropping any conflicts/dependencies. Signed-off-by: Sergiu Moga <sergiu@unikraft.io> Reviewed-by: Razvan Virtan <virtanrazvan@gmail.com> Reviewed-by: Marco Schlumpp <marco@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GitHub-Closes: unikraft#1113

This causes the PCI driver to not initialize on certain systems. The check should uses `==` instead. Signed-off-by: Kha Dinh <khadinh@g.skku.edu> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Reviewed-by: Eduard Vintilă <eduard.vintila47@gmail.com> Reviewed-by: Stefan Jumarea <stefanjumarea02@gmail.com> GitHub-Closes: unikraft#1110

When setting `V=0`, we expect the verbosity level to be reduced, and only the build output to be printed, not the build commands. Set `Q = @` by default, in case `V != 0` is set, set `Q` appropriately. Signed-off-by: Stefan Jumarea <stefanjumarea02@gmail.com> GitHub-Fixes: unikraft#676 Reviewed-by: Razvan Virtan <virtanrazvan@gmail.com> Reviewed-by: Robert Kuban <robert.kuban@opensynergy.com> Reviewed-by: Maria Sfiraiala <maria.sfiraiala@gmail.com> Approved-by: Razvan Deaconescu <razvand@unikraft.io> GitHub-Closes: unikraft#1066

Currently `9pfs` returns 0 for any `ioctl` request (except for `TIOCGWINSZ`), which leads to multiple cases of files being interpreted as terminals. You can see this problem when running python scripts, since python uses `ioctl` calls to check if it should open up the interactive interpretor or execute a script. Same thing could happen for other applications that rely on `ioctl` operations being properly executed, so the best way to go would be return `ENOTSUP` (as is used to happen before commit 28d0edf). If the `ioctl` call always return `ENOTSUP`, the Ruby binary compatibility application will fail, since it tries to set O_ASYNC and breaks on error, even if it does not directly depends on that, so for now add a special case for that. Signed-off-by: Stefan Jumarea <stefanjumarea02@gmail.com> Reviewed-by: Sergiu Moga <sergiu@unikraft.io> Approved-by: Simon Kuenzer <simon@unikraft.io> GitHub-Closes: unikraft#1098

Some applications (e.g. python) will use `ioctl()` calls to check if an interactive interpretor should be started or if a file should be read. `9pfs` should respond to all terminal-related calls with an `ENOTTY` error, so applications know that they are not running in an interactive environment. In order to detect the ioctl request type, add a new macro, `IOCTL_CMD_ISTYPE(cmd, type)`, which will check the corresponding bytes from the request number (an enhanced version of the Linux `_IOC_TYPE`). Signed-off-by: Stefan Jumarea <stefanjumarea02@gmail.com> Reviewed-by: Sergiu Moga <sergiu@unikraft.io> Approved-by: Simon Kuenzer <simon@unikraft.io> GitHub-Closes: unikraft#1098

We add an internal library for KASAN instrumentation. Currently, KASAN is available only under KVM x86_64. We use 16 MBits for the shadow memory and it's mapped after _end. This implementation is inspired from the mimiker operating system. Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

We reserve the first 16 MBytes of the heap for the shadow memory. Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

We disable KASAN for the KVM platform code. Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

We mark the memory as valid or invalid in the allocator. This is used by the instrumented code to check for faults in memory usage. Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

vladandrew changed the title ~~Vladb/kasan~~ Add Kernel Adress Sanitization support Apr 25, 2021

nderjung added arch/x86_64 plat/kvm Unikraft for KVM new/library This PR or Issue intends to add a new library to Unikraft lib/allocbuddy lib/ukalloc lib/ukboot labels Apr 26, 2021

vladandrew force-pushed the vladb/kasan branch from 3155227 to 9f95468 Compare June 5, 2021 18:41

vladandrew requested a review from hlef June 5, 2021 18:42

vladandrew requested review from danield20 and removed request for hlef September 21, 2021 16:20

danield20 approved these changes Sep 23, 2021

View reviewed changes

nderjung reviewed Sep 28, 2021

View reviewed changes

razvand added this to the v0.6 - Dione milestone Nov 17, 2021

unikraft-bot assigned vladandrew Nov 26, 2021

razvand assigned nderjung and unassigned vladandrew Nov 27, 2021

razvand requested a review from nderjung November 27, 2021 11:18

vladandrew force-pushed the vladb/kasan branch from 9f95468 to d9d5fbd Compare November 27, 2021 23:26

vladandrew requested a review from a team November 27, 2021 23:26

nderjung removed request for a team and nderjung November 28, 2021 11:11

nderjung requested changes Dec 1, 2021

View reviewed changes

vladandrew force-pushed the vladb/kasan branch from d9d5fbd to 3a4672a Compare December 1, 2021 18:31

vladandrew requested a review from nderjung December 1, 2021 18:40

razvand modified the milestones: v0.6 - Dione, v0.7 - Mimas Dec 2, 2021

jschlumpp and others added 14 commits October 3, 2023 15:28

vladandrew force-pushed the vladb/kasan branch from dbd073d to f5d8779 Compare October 6, 2023 09:54

vladandrew added 6 commits October 6, 2023 13:06

lib/ukboot: Reserve part of the heap for shadow memory

a8faf4a

We reserve the first 16 MBytes of the heap for the shadow memory. Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

plat/kvm: Disable KASAN for KVM platform code

bafcd0a

We disable KASAN for the KVM platform code. Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

lib/ukalloc: Disable KASAN for ukalloc

8eafaa0

Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

lib/ukallocbbuddy: Disable KASAN for ukallocbbuddy

764a61a

Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

lib/ukalloc: Instrument the allocator to support KASAN

ee31c37

We mark the memory as valid or invalid in the allocator. This is used by the instrumented code to check for faults in memory usage. Signed-off-by: Vlad-Andrei Badoiu <vlad_andrei.badoiu@upb.ro>

vladandrew force-pushed the vladb/kasan branch from f5d8779 to ee31c37 Compare October 6, 2023 13:29

razvand force-pushed the staging branch from 6759927 to bd59fe5 Compare October 20, 2023 16:36

razvand requested review from a team as code owners October 20, 2023 16:36

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Kernel Adress Sanitization support#191

Add Kernel Adress Sanitization support#191
vladandrew wants to merge 36 commits intounikraft:stagingfrom
vladandrew:vladb/kasan

vladandrew commented Apr 25, 2021 •

edited

Loading

Uh oh!

danield20 left a comment

Uh oh!

nderjung left a comment

Uh oh!

nderjung left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladandrew commented Dec 1, 2021

Uh oh!

nderjung commented Dec 2, 2021

Uh oh!

vladandrew commented Dec 2, 2021 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

Conversation

vladandrew commented Apr 25, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

danield20 left a comment

Choose a reason for hiding this comment

Uh oh!

nderjung left a comment

Choose a reason for hiding this comment

Uh oh!

nderjung left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

vladandrew commented Dec 1, 2021

Uh oh!

nderjung commented Dec 2, 2021

Uh oh!

vladandrew commented Dec 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

15 participants

vladandrew commented Apr 25, 2021 •

edited

Loading

vladandrew commented Dec 2, 2021 •

edited

Loading