[build] Add OSS Azure Pipeline Linux Build #4160

pjcollins · 2020-01-22T21:36:24Z

Introduces a new azure-pipelines-oss.yaml file which contains one job
for building on Linux. The pipeline definition which tracks this yaml
file can be found at:

https://dev.azure.com/xamarin/public/_build?definitionId=48&_a=summary

A few build system tweaks have also been applied to improve "partial"
build scenarios, and to allow us to build on the hosted Ubuntu agents.

xaprepare has been updated to generate the correct make rules when
partial JIT, HOST, and AOT ABIs are enabled. Previously, xaprepare would
would enable all ABIs in rules.mk and disregard the value of the
corresponding MSBuild properties. This has been fixed to allow for
partial build support. Our unix builds will no longer attempt to cross
compile Windows binaries by default as a result of this change.

make jenkins has been updated to improve the local build experience
when PREPARE_CI* flags are not set. We only need to ensure the -a
argument is passed to xaprepare when make jenkins is invoked locally,
we don't need to otherwise alter the behavior of xaprepare.

Linux package dependencies have been updated, and auto provisioning has
been fixed to automatically answer yes to installation prompts.

Finally, xaprepare has been updated to delete dependencies from
android-archives after they are extracted into android-toolchain
when running on an Azure Pipelines Hosted agent. This was done to avoid
disk space limitations on these hosted environments.

Fixes: #2009
Fixes: #4116

pjcollins · 2020-01-22T21:40:51Z

I believe packaging is still busted in the same way as our current Jenkins build. That failure will need to be investigated.

pjcollins · 2020-01-22T21:41:02Z

/azp run Xamarin.Android-OSS

azure-pipelines · 2020-01-22T21:41:07Z

No pipelines are associated with this pull request.

build-tools/xaprepare/xaprepare/ConfigAndData/Dependencies/Linux.DebianCommon.cs

pjcollins · 2020-01-24T18:54:27Z

Looks like we've got artifacts again 🎉. Could someone (maybe @grendello 😄) take a quick look at these packages to make sure they're functional?

https://dev.azure.com/xamarin/public/_build/results?buildId=13042&view=artifacts&type=publishedArtifacts

janusw · 2020-01-25T16:45:56Z

Looks like we've got artifacts again tada.

Woohoo! 😃

Thanks for working on this. 👍

Could someone (maybe @grendello smile) take a quick look at these packages to make sure they're functional?

I just downloaded the linux packages to an Ubuntu 18.04 system, extracted the .tar.xz file (btw, what's the difference between this one and the .tar.bz2 archive?), and tried to build XA's HelloWorld sample with it, like this:

~/github/xamarin-android/samples/HelloWorld$ ~/xamarin.android-oss_10.2.99.110/bin/Release/bin/xabuild /p:AndroidSdkDirectory=/home/janus/android-toolchain/sdk HelloWorld.csproj

... which unfortunately fails with:

/home/janus/xamarin.android-oss_10.2.99.110/bin/Release/lib/xamarin.android/xbuild/Xamarin/Android/Xamarin.Android.Common.targets(1343,2):
error XARLP7000: System.DllNotFoundException: lib64/libzip.so
assembly:<unknown assembly> type:<unknown type> member:(null)
[/home/janus/github/xamarin-android/samples/HelloWorld/HelloWorld.csproj]

This looks exactly like the problem which I reported as #4100 a few weeks ago. Luckily it was already fixed (with 3ae6f25), but apparently your branch does not yet include that fix.

Therefore I'm hopeful that the problem will be gone once you rebase your branch on current master. Could you please try that?

janusw · 2020-01-26T11:48:46Z

Could someone (maybe @grendello smile) take a quick look at these packages to make sure they're functional?

I just downloaded the linux packages to an Ubuntun 18.04 system, extracted the .tar.xz file (btw, what's the difference between this one and the .tar.bz2 archive?)

After some further investigation, I actually found a difference, and it's even related to the problems I saw:

xamarin.android-oss_10.2.99.110.tar.xz does not contain any .so files at all
xamarin.android-oss-v10.2.99.110_Linux-x86_64_pjcollins_oss-ubuntu_770c77f0-Release.tar.bz2 has lots of them, so it seems that this is the proper binary package for Linux. I'm not quite sure what the .tar.xz is, then.

and tried to build XA's HelloWorld sample with it, like this:

~/github/xamarin-android/samples/HelloWorld$ ~/xamarin.android-oss_10.2.99.110/bin/Release/bin/xabuild /p:AndroidSdkDirectory=/home/janus/android-toolchain/sdk HelloWorld.csproj

Just retried this with the .tar.bz2 instead of the .tar.xz archive, and this indeed seems to work. So, thanks again for the great work!

Therefore I'm hopeful that the problem will be gone once you rebase your branch on current master. Could you please try that?

Apparently my problems were unrelated to #4100 after all, but rebasing will not hurt either :)

janusw · 2020-01-26T16:03:36Z

and tried to build XA's HelloWorld sample with it, like this:
~/github/xamarin-android/samples/HelloWorld$ ~/xamarin.android-oss_10.2.99.110/bin/Release/bin/xabuild /p:AndroidSdkDirectory=/home/janus/android-toolchain/sdk HelloWorld.csproj
Just retried this with the .tar.bz2 instead of the .tar.xz archive, and this indeed seems to work. So, thanks again for the great work!

After I got this working on 18.04, I was curious to see whether the Azure build would also work on Ubuntu 16.04.

When trying this, I again got the same error as before (System.DllNotFoundException: lib64/libzip.so), but this time with the correct tarball that includes the .so files.

I first double-checked that libzip.so is actually there and then looked at its dependencies:

~/xamarin.android-oss-v10.2.99.110_Linux-x86_64/bin/Release/lib/xamarin.android/xbuild/Xamarin/Android/lib64$ ldd libzip.so 
	linux-vdso.so.1 =>  (0x00007ffd63785000)
	libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f5c5dd7e000)
	libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f5c5db6e000)
	libcrypto.so.1.1 => not found
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5c5d7a4000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f5c5e1b8000)

So, this now really looks like #4100: The libzip.so that is included in the XA build depends on libcrypto.so.1.1, but Ubuntu 16.04 only provides libcrypto.so.1.0.0.

I am confident that this error will go away once the branch is rebased on current master 😊 (and I'll be happy to verify this once you do the rebase ...)

Akademiksdc · 2020-01-27T13:04:56Z

@janusw seems that on the Ubuntu 16.04 still no success,
The following error is still there:
error XARLP7000: System.DllNotFoundException: lib64/libzip.so assembly:<unknown assembly>

pjcollins · 2020-01-27T17:32:14Z

Hmm, we should be building a merge commit and therefore the artifacts from last Friday should have contained the fixes mentioned above. We also probably don't need both tar files so I will try to clean that up. Let me try to get another build running today after rebasing and see if that clears things up.

These items are already included in LinuxDebianCommon. The i386 versions also no longer appear to exist?

We shouldn't set all CI related xaprepare flags when building locally. We also shouldn't be passing `-a` when the "PR" build CI flags are set.

Should hopefully fix the following Linux packaging error: dh_clideps: Error: Missing shlibs entry: libmono-btls-shared.so or libmono-btls-shared for: System.dll!

janusw · 2020-01-28T07:31:30Z

Hopefully this one works... https://dev.azure.com/xamarin/public/_build/results?buildId=13105&view=results

Yes, this one seems to work now, even on Ubuntu 16.04 (using the .tar.bz2 package). Nice work! I hope this PR can be merged to master (looks like you have an approval already).

I believe the .tar.xz archive is an artifact from the dpkg-buildpackage command and can be ignored.

Thanks for the explanation. So, if this one is missing the .so files, does that mean they are also missing in the .deb file? (I will try the deb package tonight.)

janusw · 2020-01-28T10:05:07Z

One more comment here: I tried to use xabuild on Ubuntu to build a project that is based on .NET Standard 2.1:

  <PropertyGroup>
    <TargetFramework>netstandard2.1</TargetFramework>
    [...]
  </PropertyGroup>

This fails with:

/usr/lib/mono/msbuild/Current/bin/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.FrameworkReferenceResolution.targets(262,5):
error NETSDK1073: The FrameworkReference 'NETStandard.Library' was not recognized

However it works with msbuild on Linux (and also e.g. with Visual Studio on MacOS). I have Ubuntu 16.04 with dotnet-sdk-3.1 and mono 6.4.0.

I can work around it by downgrading to .NET Standard 2.0.

Is this a problem with the XA build, or am I just using it incorrectly? Note that I have simply unpacked the tarball to my home folder and run it from there.

pjcollins · 2020-01-28T16:27:42Z

@janusw thanks for your time and testing on this, much appreciated!

Thanks for the explanation. So, if this one is missing the .so files, does that mean they are also missing in the .deb file? (I will try the deb package tonight.)

These .so files should be in the .deb, at least that is what I saw yesterday when extracting locally.

I tried to use xabuild on Ubuntu to build a project that is based on .NET Standard 2.1:

We don't recommend using xabuild for building anything other than Xamarin.Android projects. We may try to move away from this build tool in the coming year or so as well. Are you trying to build a standalone NET Standard project, or is this a failure in a Xamarin.Android project with a NET Standard reference? In either case, I'd recommend installing the .deb and using the msbuild provided by mono.

Finally as a note, I'm considering removing the upload of the .tar.bz2 file from this build, as recent Ubuntu related build issues should hopefully now be resolved.

janusw · 2020-01-28T20:03:40Z

@janusw thanks for your time and testing on this, much appreciated!

You're very welcome. I highly appreciate the work you're doing here as well. It's great to see that XA is working on Linux again.

By now I have verified that the .deb package installs cleanly on Ubuntu 16.04 and 18.04 and makes it possible to build an apk file of the HelloWorld sample via msbuild:

~/github/xamarin-android/samples/HelloWorld$
msbuild /p:AndroidSdkDirectory=/home/janus/android-toolchain/sdk /t:SignAndroidPackage HelloWorld.csproj

So, IMHO the Linux build works very well and is definitely ready for master.

Are you trying to build a standalone NET Standard project, or is this a failure in a Xamarin.Android project with a NET Standard reference?

This is a NET standard project that is included in a cross-platform solution, where it's referenced by an Android and an iOS project. Here I was actually trying to compile the NET standard project by itself (for starters).

I'll try tomorrow whether I can get this working with msbuild.

pjcollins · 2020-01-28T21:22:31Z

The test failures here look unrelated, the two failures are caused by 1. a NuGet restore (which looks like a potential test parallelization edge case), and 2. the somewhat rare keytool error we've seen elsewhere in the past.

janusw · 2020-01-29T07:30:22Z

By now I have verified that the .deb package installs cleanly on Ubuntu 16.04 and 18.04 and makes it possible to build an apk file of the HelloWorld sample via msbuild:

Actually I just noticed that the installation on 16.04 was not fully clean:

$ sudo dpkg -i xamarin.android-oss_10.2.99.180_amd64.deb 
Selecting previously unselected package xamarin.android-oss.
(Reading database ... 487940 files and directories currently installed.)
Preparing to unpack xamarin.android-oss_10.2.99.180_amd64.deb ...
Unpacking xamarin.android-oss (10.2.99.180) ...
dpkg: dependency problems prevent configuration of xamarin.android-oss:
 xamarin.android-oss depends on libc6 (>= 2.27); however:
  Version of libc6:amd64 on system is 2.23-0ubuntu11.

dpkg: error processing package xamarin.android-oss (--install):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Errors were encountered while processing:
 xamarin.android-oss

Apparently the installation as such went through, but the configuration was canceled due to this dependency problem. The HelloWorld compiles fine nonetheless.

Could you comment on what the configuration process is doing exactly? Is this an actual problem or can one just ignore the error message?

omarbekov · 2020-01-29T08:52:26Z

I can also confirm that I can build apps successfully.
Debugging with IDE doesn't work though (does not stop at breakpoints).
But I guess I have to use debug builds for that.

janusw · 2020-01-29T11:22:58Z

I can also confirm that I can build apps successfully.

Which OS are you on?

I have also managed to build a full-fledged app by now, not just the simple HelloWorld example.

The only thing I noticed is that I cannot use .NET Standard 2.1 on Linux, but I assume this is unrelated to XA. May be due to my Mono version (6.4)? I'll try a newer one soon.

Debugging with IDE doesn't work though (does not stop at breakpoints).

Which IDE? MonoDevelop?

janusw · 2020-01-29T11:26:59Z

The only thing I noticed is that I cannot use .NET Standard 2.1 on Linux, but I assume this is unrelated to XA. May be due to my Mono version (6.4)? I'll try a newer one soon.

In particular I see the following errors when running nuget restore:

NU1201: Project XXX is not compatible with monoandroid10.0 (MonoAndroid,Version=v10.0).
Project XXX supports: netstandard2.1 (.NETStandard,Version=v2.1)

Can anyone tell me where this limitation comes from? Is this due to Xamarin.Android or Mono?

janusw · 2020-01-29T12:06:54Z

The only thing I noticed is that I cannot use .NET Standard 2.1 on Linux, but I assume this is unrelated to XA. May be due to my Mono version (6.4)? I'll try a newer one soon.

In particular I see the following errors when running nuget restore:
NU1201: Project XXX is not compatible with monoandroid10.0 (MonoAndroid,Version=v10.0).
Project XXX supports: netstandard2.1 (.NETStandard,Version=v2.1)
Can anyone tell me where this limitation comes from? Is this due to Xamarin.Android or Mono?

In fact these errors went away once I upgraded Mono from 6.4 to 6.8. The Mono release notes actually claim support for .NET Standard 2.1 already in 6.4. Either this is plain wrong, or there is at least still a bug there :(

In any case, 6.8.0 works nicely and the problem was not related to XA after all. 🎉

janusw · 2020-01-29T13:40:51Z

Actually I just noticed that the installation on 16.04 was not fully clean:
$ sudo dpkg -i xamarin.android-oss_10.2.99.180_amd64.deb 
Selecting previously unselected package xamarin.android-oss.
(Reading database ... 487940 files and directories currently installed.)
Preparing to unpack xamarin.android-oss_10.2.99.180_amd64.deb ...
Unpacking xamarin.android-oss (10.2.99.180) ...
dpkg: dependency problems prevent configuration of xamarin.android-oss:
 xamarin.android-oss depends on libc6 (>= 2.27); however:
  Version of libc6:amd64 on system is 2.23-0ubuntu11.

dpkg: error processing package xamarin.android-oss (--install):
 dependency problems - leaving unconfigured
Processing triggers for libc-bin (2.23-0ubuntu11) ...
Errors were encountered while processing:
 xamarin.android-oss
Apparently the installation as such went through, but the configuration was canceled due to this dependency problem. The HelloWorld compiles fine nonetheless.

Could you comment on what the configuration process is doing exactly? Is this an actual problem or can one just ignore the error message?

From my tests so far, I would conclude that the XA .deb package is fully working on Ubuntu 16.04, despite the warnings about different libc version. If this is the case, I would suggest to remove the dependency on libc in the .deb package, if possible.

Even though one can install the .deb, the dependency problem seriously screws up apt. It's kind of hard to install further packages, because apt always complains about unmet dependencies and wants to uninstall xamarin.android to resolve that :(

pjcollins · 2020-01-29T18:12:41Z

Actually I just noticed that the installation on 16.04 was not fully clean

@janusw Is this an issue only with 16.04? I imagine this is a result of us building on 18.04, but I am not entirely sure. If this only occurs on 16.04 I think it's likely not something that we will try to resolve unfortunately as the OS version itself is pretty close to being in it's last year of support.

janusw · 2020-01-29T18:46:05Z

Actually I just noticed that the installation on 16.04 was not fully clean

@janusw Is this an issue only with 16.04?

Yes, 16.04 only.

I imagine this is a result of us building on 18.04, but I am not entirely sure.

Yes, it is definitely due to the difference in libc version between 16.04 and 18.04.

If this only occurs on 16.04 I think it's likely not something that we will try to resolve unfortunately as the OS version itself is pretty close to being in it's last year of support.

It's actually not a big issue. Everything seems to work fine on 16.04. The only problem is that the package manager thinks that XA requires libc version 2.27 (with which is was built), although it seems to work with 2.23 as well (for all I know).

So maybe one could just remove the dependency on libc from the .deb package altogether, which should not be too hard, I think.

It seems one can even fix up the dependencies of a package afterwards: https://coderwall.com/p/hes3ha/change-the-dependencies-of-a-deb-package

janusw · 2020-01-29T19:18:55Z

I think it's likely not something that we will try to resolve unfortunately as the OS version itself is pretty close to being in it's last year of support.

I understand if this is not a priority for you right now, and I'm perfectly fine with merging this PR without fixing the .deb dependencies.

But note that this not only affects Ubuntu 16.04, but also Debian itself (stretch has libc 2.24 and will be supported until 2022) and other Debian-based systems. So fixing this might affect quite a few possible users.

For me personally this is not a big deal, since I will probably move to 18.04 and/or building XA myself soon.

omarbekov · 2020-01-30T03:25:22Z

@janusw Jetbrains Rider 2019.2.2 on Xubuntu 18.04

dellis1972 · 2020-01-30T09:49:06Z

I can also confirm that I can build apps successfully.
Debugging with IDE doesn't work though (does not stop at breakpoints).
But I guess I have to use debug builds for that.

@lordofffarm this is just the open source parts of Xamarin.Android as far as I can see. So it probably does not include the required tasks/targets needed to setup the debugger etc on android. Having. debug build of this package would not help in that case.

omarbekov · 2020-01-30T10:14:55Z

probably does not include the required tasks/targets needed to setup the debugger

@dellis1972 Thanks for the reply.
Do you know whether these "tasks/targets" are going to be included in builds, when this PR gets merged?
I can definitely debug with the last successful build v9.2.99.172 from Jenkins.
https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-linux/2194/Azure/

pjcollins · 2020-01-30T15:56:48Z

@lordofffarm the packages you've linked from Jenkins do not contain debug tasks/targets either, so I'm not exactly sure how this was working for you before.

omarbekov · 2020-01-31T05:24:24Z

@pjcollins Thanks for your work, man.
I just tried to debug with both Debug and Release builds from v9.2.99.172 and debugging does indeed work.
It doesn't work with your last build though, unfortunately.
I even tried updating Rider to the latest version 2019.3.1.
The version of Mono is 6.8.0.96.

omarbekov · 2020-01-31T07:54:31Z

Ok, I got debugging to work now.
In the Debug configuration of our app "AndroidLinkMode" was set to "SdkOnly".
I changed it to "None" and now it stops at breakpoints.

janusw · 2020-01-31T09:46:03Z

Ok, I got debugging to work now.

Great, sounds like everybody's happy 😃 so hopefully this can be merged ...

pjcollins · 2020-01-31T15:36:32Z

I think we're waiting on a final review or two here but it should hopefully land soon.

grendello · 2020-02-03T08:43:24Z

Actually I just noticed that the installation on 16.04 was not fully clean

@janusw Is this an issue only with 16.04? I imagine this is a result of us building on 18.04, but I am not entirely sure. If this only occurs on 16.04 I think it's likely not something that we will try to resolve unfortunately as the OS version itself is pretty close to being in it's last year of support.

Yes, this is the reason. It could be solved by explicitly specifying the accepted/supported libc versions in debian/control, however that may leave room for errors and possibly be against the Debian packaging policy?

grendello · 2020-02-03T08:46:54Z

LGTM :) @pjcollins I'm going to merge and ignore the libc dependency issue, we can address that in a separate PR.

pjcollins requested review from grendello and jonpryor as code owners January 22, 2020 21:36

pjcollins added the do-not-merge PR should not be merged. label Jan 22, 2020

grendello reviewed Jan 22, 2020

View reviewed changes

build-tools/xaprepare/xaprepare/ConfigAndData/Dependencies/Linux.DebianCommon.cs Show resolved Hide resolved

pjcollins requested a review from directhex as a code owner January 23, 2020 17:11

pjcollins mentioned this pull request Jan 24, 2020

Improve xaprepare to dynamically install dependencies needed for the build configuration #4173

Closed

pjcollins removed the do-not-merge PR should not be merged. label Jan 24, 2020

dellis1972 approved these changes Jan 27, 2020

View reviewed changes

pjcollins and others added 13 commits January 27, 2020 12:52

Pass yes to apt-get installation prompts

078323f

Remove duplicate ubuntu tools

a7a3a68

These items are already included in LinuxDebianCommon. The i386 versions also no longer appear to exist?

[build] Fix local/PR make jenkins invocations

74009f6

We shouldn't set all CI related xaprepare flags when building locally. We also shouldn't be passing `-a` when the "PR" build CI flags are set.

Write only the enabled ABIs to rules.mk

de5037c

Add nuget and devscripts deb packages

c73c32f

Add OSS pipeline file w/ linux build

f9fc852

Add cli-common-dev deb package

98f506a

Clean up toolchain cache after provisioning on hosted machines

531e0c5

[build] Add libmono-btls-shared to exclusion list

ac01f02

Should hopefully fix the following Linux packaging error: dh_clideps: Error: Missing shlibs entry: libmono-btls-shared.so or libmono-btls-shared for: System.dll!

Update libmono-btls-shared exclusion to not include extension

78325a9

[ci] Install mono preview before linx build

d1cadef

Update README.md to include OSS Linux build status

0a81242

Only include tar.bz2 linux artifact

3695798

janusw mentioned this pull request Jan 29, 2020

Support Xamarin.Android on linux mono/monodevelop#8712

Open

grendello approved these changes Feb 3, 2020

View reviewed changes

grendello merged commit 2c24731 into master Feb 3, 2020

pjcollins deleted the pjcollins_oss-ubuntu branch February 18, 2020 16:08

brendanzagaeski added this to the In Preview: 10.3 (d16-6) milestone Mar 17, 2020

github-actions bot locked and limited conversation to collaborators Jan 24, 2024

[build] Add OSS Azure Pipeline Linux Build #4160

[build] Add OSS Azure Pipeline Linux Build #4160

Uh oh!

Conversation

pjcollins commented Jan 22, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pjcollins commented Jan 22, 2020

Uh oh!

pjcollins commented Jan 22, 2020

Uh oh!

azure-pipelines bot commented Jan 22, 2020

Uh oh!

Uh oh!

pjcollins commented Jan 24, 2020

Uh oh!

janusw commented Jan 25, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janusw commented Jan 26, 2020

Uh oh!

janusw commented Jan 26, 2020

Uh oh!

Akademiksdc commented Jan 27, 2020

Uh oh!

pjcollins commented Jan 27, 2020

Uh oh!

janusw commented Jan 28, 2020

Uh oh!

janusw commented Jan 28, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pjcollins commented Jan 28, 2020

Uh oh!

janusw commented Jan 28, 2020

Uh oh!

pjcollins commented Jan 28, 2020

Uh oh!

janusw commented Jan 29, 2020

Uh oh!

omarbekov commented Jan 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

janusw commented Jan 29, 2020

Uh oh!

janusw commented Jan 29, 2020

Uh oh!

janusw commented Jan 29, 2020

Uh oh!

janusw commented Jan 29, 2020

Uh oh!

pjcollins commented Jan 29, 2020

Uh oh!

janusw commented Jan 29, 2020

Uh oh!

janusw commented Jan 29, 2020

Uh oh!

omarbekov commented Jan 30, 2020

Uh oh!

dellis1972 commented Jan 30, 2020

Uh oh!

omarbekov commented Jan 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pjcollins commented Jan 30, 2020

Uh oh!

omarbekov commented Jan 31, 2020

Uh oh!

omarbekov commented Jan 31, 2020

Uh oh!

janusw commented Jan 31, 2020

Uh oh!

pjcollins commented Jan 31, 2020

Uh oh!

grendello commented Feb 3, 2020

Uh oh!

grendello commented Feb 3, 2020

Uh oh!

Reviewers

Assignees

pjcollins commented Jan 22, 2020 •

edited

Loading

janusw commented Jan 25, 2020 •

edited

Loading

janusw commented Jan 28, 2020 •

edited

Loading

omarbekov commented Jan 29, 2020 •

edited

Loading

omarbekov commented Jan 30, 2020 •

edited

Loading