FreeBSD needs CI-build

[josteink]

With FreeBSD now having a working build (yay!), we need to setup an official CI-build so we can avoid breaking it in the future.

@richlander You said you could help make this happen. Do you need anything from the FreeBSD port-team to get this going?

[richlander]

@janhenke was asking the same thing in the gitter. @mmitche is looking into it. There is no stock image for FreeBSD in the Azure gallery, so it will require a little work on our side.

Hurrah!! Congrats, FreeBSD Port Team. Nice job. I guess, however, that this is just the first milestone.

[richlander]

@janhenke was asking the same thing in the gitter. @mmitche is looking into it. There is no stock image for FreeBSD in the Azure gallery, so it will require a little work on our side.

Hurrah!! Congrats, FreeBSD Port Team. Nice job. I guess, however, that this is just the first milestone.

[ajensenwaud]

Hi all,

In the interim I have set up a Jenkins CI server in our build environment. You can access it here:

http://freebsd-frankfurt.zbsd.org/jenkins/

This should do for now -- happy to help out on the .NET CoreCLR side if you need help setting up a FreeBSD Azure instance with Jenkins, etc. I can set it up very quickly.

If anyone

[ajensenwaud]

Hi all,

In the interim I have set up a Jenkins CI server in our build environment. You can access it here:

http://freebsd-frankfurt.zbsd.org/jenkins/

This should do for now -- happy to help out on the .NET CoreCLR side if you need help setting up a FreeBSD Azure instance with Jenkins, etc. I can set it up very quickly.

If anyone

[koobs]

👍

[koobs]

👍

[ghuntley]

To the entire port team, congratulations:

😍 @janhenke, @josteink, @ajensenwaud, @kangaroo, @ghuntley, @richlander 😍

@richlander / @mmitche there is indeed a stock image available in the Azure Gallery which is hosted by MSFT Open Technologies that works perfectly. That was the image used to deploy portteam.cloudapp.net

https://vmdepot.msopentech.com/Vhd/Show?vhdId=49971&version=51112

Unfortunately NetBSD and OpenBSD however do not currently work on Azure.

https://wiki.netbsd.org/projects/project/netbsd_on_microsoft_azure/
http://feedback.azure.com/forums/216843-virtual-machines/suggestions/6389839-openbsd-virtual-machine

Here is the operating system and azure-cli configuration that was used to build portteam.cloudapp.net:

https://gist.github.com/ghuntley/cd56a682b0dfb0c626e5

[ghuntley]

To the entire port team, congratulations:

😍 @janhenke, @josteink, @ajensenwaud, @kangaroo, @ghuntley, @richlander 😍

@richlander / @mmitche there is indeed a stock image available in the Azure Gallery which is hosted by MSFT Open Technologies that works perfectly. That was the image used to deploy portteam.cloudapp.net

https://vmdepot.msopentech.com/Vhd/Show?vhdId=49971&version=51112

Unfortunately NetBSD and OpenBSD however do not currently work on Azure.

https://wiki.netbsd.org/projects/project/netbsd_on_microsoft_azure/
http://feedback.azure.com/forums/216843-virtual-machines/suggestions/6389839-openbsd-virtual-machine

Here is the operating system and azure-cli configuration that was used to build portteam.cloudapp.net:

https://gist.github.com/ghuntley/cd56a682b0dfb0c626e5

[richlander]

The HN visibility is unexpected: https://news.ycombinator.com/item?id=9431368.

For the sake of history, here is the original "FreeBSD Port Team: Volunteers needed" issue #455.

You can see all of the FreeBSD merged/closed issues and PRs here (although naturally some are still open):

PRs: https://github.com/dotnet/coreclr/pulls?q=label%3AFreeBSD+is%3Aclosed+is%3Apr
Issues: https://github.com/dotnet/coreclr/issues?q=label%3AFreeBSD+is%3Aclosed+is%3Aissue

Congratulations to the team. Excellent work.

[richlander]

The HN visibility is unexpected: https://news.ycombinator.com/item?id=9431368.

For the sake of history, here is the original "FreeBSD Port Team: Volunteers needed" issue #455.

You can see all of the FreeBSD merged/closed issues and PRs here (although naturally some are still open):

PRs: https://github.com/dotnet/coreclr/pulls?q=label%3AFreeBSD+is%3Aclosed+is%3Apr
Issues: https://github.com/dotnet/coreclr/issues?q=label%3AFreeBSD+is%3Aclosed+is%3Aissue

Congratulations to the team. Excellent work.

[blackdwarf]

Very cool, very cool, this is some really good work. 👍

[blackdwarf]

Very cool, very cool, this is some really good work. 👍

[ghost]

This is amazing 😢

Good job!

[ghost]

This is amazing 😢

Good job!

[brunolauze]

What is the difference between coreclr and mono? Does coreclr directly from Microsoft is poised to be outperforming mono (which is now built with the same coreclr or almost)

[brunolauze]

What is the difference between coreclr and mono? Does coreclr directly from Microsoft is poised to be outperforming mono (which is now built with the same coreclr or almost)

[jgowdy]

Well done everyone, this is a huge milestone.

[jgowdy]

Well done everyone, this is a huge milestone.

[richlander]

@brunolauze CoreCLR is Microsoft's implementation of .NET, for use in its commercial products, like ASP.NET 5. These products are open source, but we approach them as commercial products as we always have (e.g. support, reliability). What's new is the open source community engagement (like responding to your comment 😉).

We're not working on "outperforming Mono" as a goal. We have various goals around .NET Core itself. If someone concludes that CoreCLR outperforms Mono at some point, that's fine, but not a goal of its own. It is fair to say that some people using Mono now will start using CoreCLR when it matures on Linux and OS X. It is also fair to say that we expect new people to come to .NET with the increase in excitement and product choice (e.g. ASP.NET 5 on Linux). If they choose Mono over .NET Core, awesome! There is no competition with Mono of any kind.

@migueldeicaza and friends did a great job of creating and maintaining the open source .NET Runtime for well over a decade. We commend them for that. 👏 🙇 We're glad to have them as partners now that we're the newcomers in .NET OSS.

[richlander]

@brunolauze CoreCLR is Microsoft's implementation of .NET, for use in its commercial products, like ASP.NET 5. These products are open source, but we approach them as commercial products as we always have (e.g. support, reliability). What's new is the open source community engagement (like responding to your comment 😉).

We're not working on "outperforming Mono" as a goal. We have various goals around .NET Core itself. If someone concludes that CoreCLR outperforms Mono at some point, that's fine, but not a goal of its own. It is fair to say that some people using Mono now will start using CoreCLR when it matures on Linux and OS X. It is also fair to say that we expect new people to come to .NET with the increase in excitement and product choice (e.g. ASP.NET 5 on Linux). If they choose Mono over .NET Core, awesome! There is no competition with Mono of any kind.

@migueldeicaza and friends did a great job of creating and maintaining the open source .NET Runtime for well over a decade. We commend them for that. 👏 🙇 We're glad to have them as partners now that we're the newcomers in .NET OSS.

[mmitche]

@ghuntley Thanks, found it...creating it now. After I get it working i'll hook it up to the main CI. ETA is end of day.

[mmitche]

@ghuntley Thanks, found it...creating it now. After I get it working i'll hook it up to the main CI. ETA is end of day.

[mmitche]

I have a working build. I am working on the rest of the pieces (joining the Jenkins master automatically for instance).

Btw, I saw a single failure in the pal tests. I think it's the same as the one seen OSX during the bring-up. Known?

[mmitche]

I have a working build. I am working on the rest of the pieces (joining the Jenkins master automatically for instance).

Btw, I saw a single failure in the pal tests. I think it's the same as the one seen OSX during the bring-up. Known?

[korczis]

Nice work gentleman!

👍

[korczis]

Nice work gentleman!

👍

[janvorli]

@mmitche What is this failure?

[janvorli]

@mmitche What is this failure?

[mmitche]

On FreeBSD: FAILED: threading/WaitForSingleObject/WFSOExSemaphoreTest/paltest_waitforsingleobject_wfsoexsemaphoretest. Exit code: 1

[mmitche]

On FreeBSD: FAILED: threading/WaitForSingleObject/WFSOExSemaphoreTest/paltest_waitforsingleobject_wfsoexsemaphoretest. Exit code: 1

[brunolauze]

@richlander thanks for the great summary!

[brunolauze]

@richlander thanks for the great summary!

[richlander]

NP, @brunolauze. I see you are from PQ. I was in Montreal a couple weeks ago.

[richlander]

NP, @brunolauze. I see you are from PQ. I was in Montreal a couple weeks ago.

[josteink]

@mmitche Nor really "known" per se. We were just happy to have a build which finally went to 100% and yielded a proper build-output.

We've yet to investigate what works and not when executing .net-code via corerun. We still got some way to go :)

[josteink]

@mmitche Nor really "known" per se. We were just happy to have a build which finally went to 100% and yielded a proper build-output.

We've yet to investigate what works and not when executing .net-code via corerun. We still got some way to go :)

[brunolauze]

@richlander Mtl is great, I am in Quebec City. I was trying to understand the coreclr api to be able to realize a core clr c++ lib like the tentative for mono https://github.com/brunolauze/MonoNative . What do you think?

[brunolauze]

@richlander Mtl is great, I am in Quebec City. I was trying to understand the coreclr api to be able to realize a core clr c++ lib like the tentative for mono https://github.com/brunolauze/MonoNative . What do you think?

[richlander]

@brunolauze A good issue. We shouldn't continue the conversation on this issue. Either file a separate issue or start a conversation on gitter: https://gitter.im/dotnet/coreclr. Both is probably best.

[richlander]

@brunolauze A good issue. We shouldn't continue the conversation on this issue. Either file a separate issue or start a conversation on gitter: https://gitter.im/dotnet/coreclr. Both is probably best.

[richlander]

FYI: The official CoreCLR on FreeBSD build instructions have now been posted: https://github.com/dotnet/coreclr/blob/master/Documentation/README.md#build-coreclr-from-source. Was PR #800, by @janhenke.

[richlander]

FYI: The official CoreCLR on FreeBSD build instructions have now been posted: https://github.com/dotnet/coreclr/blob/master/Documentation/README.md#build-coreclr-from-source. Was PR #800, by @janhenke.

[mmitche]

@ghuntley Really close here. One issue though...for some reason Azure isn't setting the hostname on the target vm (even though it's set up to do so). Have you seen this?

[mmitche]

@ghuntley Really close here. One issue though...for some reason Azure isn't setting the hostname on the target vm (even though it's set up to do so). Have you seen this?

[jgowdy]

What's the situation with the FreeBSD build and the PTHREAD_PROCESS_SHARED support described at the top of shmemory.cpp? Is the workaround sufficient, or was better support added in newer versions of FreeBSD? Is there a performance issue there? https://github.com/dotnet/coreclr/blob/cbf46fb0b6a0b209ed1caf4a680910b383e68cba/src/pal/src/shmemory/shmemory.cpp

[jgowdy]

What's the situation with the FreeBSD build and the PTHREAD_PROCESS_SHARED support described at the top of shmemory.cpp? Is the workaround sufficient, or was better support added in newer versions of FreeBSD? Is there a performance issue there? https://github.com/dotnet/coreclr/blob/cbf46fb0b6a0b209ed1caf4a680910b383e68cba/src/pal/src/shmemory/shmemory.cpp

[janhenke]

If you want to set it manually, the hostname is defined in /etc/rc.conf

[janhenke]

If you want to set it manually, the hostname is defined in /etc/rc.conf

[ghuntley]

@mmitche yep that happened for me too just set it manually.

Jump in Gitter and we will rosetta you the way through the sysadmin differences - https://gitter.im/dotnet/coreclr

[ghuntley]

@mmitche yep that happened for me too just set it manually.

Jump in Gitter and we will rosetta you the way through the sysadmin differences - https://gitter.im/dotnet/coreclr

[mmitche]

yeah, I can do that. However we typically have changes on the machines that require us to cycle all vm's to a new image fairly frequently. So I have the system set up to launch Jenkins at startup. It needs to be able to know the hostname to get the Jenkins jnlp url though.

[mmitche]

yeah, I can do that. However we typically have changes on the machines that require us to cycle all vm's to a new image fairly frequently. So I have the system set up to launch Jenkins at startup. It needs to be able to know the hostname to get the Jenkins jnlp url though.

[josteink]

@jgowdy I would have to say we're still at "untested grounds". We have a build completing without a failure-code. We have in no way said we have a production-ready build at this point.

As for you question, It seems that things have improved since the time of shmemory.cpp's writing:

josteink@freebsd-frankfurt:~/build/coreclr % man pthread_rwlockattr_setpshared 3
PTHREAD_RWLOCKATTR_... FreeBSD Library Functions Manual PTHREAD_RWLOCKATTR_...

NAME
     pthread_rwlockattr_setpshared -- set the process shared attribute

LIBRARY
     POSIX Threads Library (libpthread, -lpthread)

SYNOPSIS
     #include <pthread.h>

     int
     pthread_rwlockattr_setpshared(pthread_rwlockattr_t *attr, int pshared);

DESCRIPTION
     The pthread_rwlockattr_setpshared() function sets the process shared
     attribute of attr to the value referenced by pshared.  The pshared argu-
     ment may be one of two values:

     PTHREAD_PROCESS_SHARED   Any thread of any process that has access to the
                              memory where the read/write lock resides can
                              manipulate the lock.

     PTHREAD_PROCESS_PRIVATE  Only threads created within the same process as
                              the thread that initialized the read/write lock
                              can manipulate the lock.  This is the default
                              value.

Beyond that I don't think I'm qualified to provide an answer. Maybe someone else on the team has deeper insights?

[josteink]

@jgowdy I would have to say we're still at "untested grounds". We have a build completing without a failure-code. We have in no way said we have a production-ready build at this point.

As for you question, It seems that things have improved since the time of shmemory.cpp's writing:

josteink@freebsd-frankfurt:~/build/coreclr % man pthread_rwlockattr_setpshared 3
PTHREAD_RWLOCKATTR_... FreeBSD Library Functions Manual PTHREAD_RWLOCKATTR_...

NAME
     pthread_rwlockattr_setpshared -- set the process shared attribute

LIBRARY
     POSIX Threads Library (libpthread, -lpthread)

SYNOPSIS
     #include <pthread.h>

     int
     pthread_rwlockattr_setpshared(pthread_rwlockattr_t *attr, int pshared);

DESCRIPTION
     The pthread_rwlockattr_setpshared() function sets the process shared
     attribute of attr to the value referenced by pshared.  The pshared argu-
     ment may be one of two values:

     PTHREAD_PROCESS_SHARED   Any thread of any process that has access to the
                              memory where the read/write lock resides can
                              manipulate the lock.

     PTHREAD_PROCESS_PRIVATE  Only threads created within the same process as
                              the thread that initialized the read/write lock
                              can manipulate the lock.  This is the default
                              value.

Beyond that I don't think I'm qualified to provide an answer. Maybe someone else on the team has deeper insights?

[josteink]

@mmitche I may be asking stupid questions, but have you checked your image against the manual procedure's outlined in this Azure blog-post here?

It might contain leads which will help you get a good image for de/re-provisioning.

[josteink]

@mmitche I may be asking stupid questions, but have you checked your image against the manual procedure's outlined in this Azure blog-post here?

It might contain leads which will help you get a good image for de/re-provisioning.

[mmitche]

yep. Everything appears to be working except for the automatic azure hostname setting.

[mmitche]

yep. Everything appears to be working except for the automatic azure hostname setting.

[mmitche]

http://dotnet-ci.cloudapp.net/job/dotnet_coreclr_freebsd_debug/1/console
http://dotnet-ci.cloudapp.net/job/dotnet_coreclr_freebsd_release/1/console

[mmitche]

http://dotnet-ci.cloudapp.net/job/dotnet_coreclr_freebsd_debug/1/console
http://dotnet-ci.cloudapp.net/job/dotnet_coreclr_freebsd_release/1/console

[mmitche]

It's alive! The build time is pretty hefty right now compared to Linux (1 hour vs. 15 minutes or so). Do you guys have any idea why?

Btw, right now the jobs pass but I'm going to hold off until Monday to integrate them into the PR/CI workflow so they don't block progress if something goes wrong.

[mmitche]

It's alive! The build time is pretty hefty right now compared to Linux (1 hour vs. 15 minutes or so). Do you guys have any idea why?

Btw, right now the jobs pass but I'm going to hold off until Monday to integrate them into the PR/CI workflow so they don't block progress if something goes wrong.

[janvorli]

@mmitche My guess would be that the FreeBSD VM or the OS itself uses just one processor core.

[janvorli]

@mmitche My guess would be that the FreeBSD VM or the OS itself uses just one processor core.

[mmitche]

it's detecting 4 processors...hm

[mmitche]

it's detecting 4 processors...hm

[ajensenwaud]

How much memory is allocated to the instance? On our Interoute instance build takes less than 10 mins.

Anders Jensen-Waud
+61 427 948 715

On 25 Apr 2015, at 08:45, Matt Mitchell notifications@github.com wrote:

it's detecting 4 processors...hm

—
Reply to this email directly or view it on GitHub.

[ajensenwaud]

How much memory is allocated to the instance? On our Interoute instance build takes less than 10 mins.

Anders Jensen-Waud
+61 427 948 715

On 25 Apr 2015, at 08:45, Matt Mitchell notifications@github.com wrote:

it's detecting 4 processors...hm

—
Reply to this email directly or view it on GitHub.

[mmitche]

@ajensenwaud It looks like it was a result of two builds hitting the machine at the same time. On the Linux machines this is okay, but on BSD it doesn't work as well. I can bump it down to one concurrent job though.

Machines have 4 procs and 14 gb ram.

[mmitche]

@ajensenwaud It looks like it was a result of two builds hitting the machine at the same time. On the Linux machines this is okay, but on BSD it doesn't work as well. I can bump it down to one concurrent job though.

Machines have 4 procs and 14 gb ram.

[ajensenwaud]

Ok we have a considerably larger build machine, which may play a role as well. :) 12 cpus and 64 GB RAM...

Anders Jensen-Waud
+61 427 948 715

On 25 Apr 2015, at 09:09, Matt Mitchell notifications@github.com wrote:

@ajensenwaud It looks like it was a result of two builds hitting the machine at the same time. On the Linux machines this is okay, but on BSD it doesn't work as well. I can bump it down to one concurrent job though.

Machines have 4 procs and 14 gb ram.

—
Reply to this email directly or view it on GitHub.

[ajensenwaud]

Ok we have a considerably larger build machine, which may play a role as well. :) 12 cpus and 64 GB RAM...

Anders Jensen-Waud
+61 427 948 715

On 25 Apr 2015, at 09:09, Matt Mitchell notifications@github.com wrote:

@ajensenwaud It looks like it was a result of two builds hitting the machine at the same time. On the Linux machines this is okay, but on BSD it doesn't work as well. I can bump it down to one concurrent job though.

Machines have 4 procs and 14 gb ram.

—
Reply to this email directly or view it on GitHub.

[josteink]

@mmitche That's odd. Building something like coreclr is fairly IO-intensive so I did some basic IO-testing on our Interroute VMs vs our more limited Azure VMs.

Although there's probably a million errors in the measuremeant technique scientifically speaking, the difference is night and day:

# Interroute
$ dd if=/dev/zero of=test bs=1M count=1000 
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.624000 secs (1,680,410,061 bytes/sec)

# Azure
$  dd if=/dev/zero of=test bs=1M count=1000 
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 130.005050 secs (8,065,656 bytes/sec)

What causes this massive difference in disk-IO, I don't know. It might be available memory available for caching. It might be the ZFS filesystem configuration (a good volume for the ZIL?). The fact of the matter is that we're getting around 250x the IO throughput on Interroute's VMware-based servers.

If we can boost the IO-throughput on our Azure-machines we should be able to bring down the build-time significantly. For a CI-server you probably want more than 8MB/s. Out of interest, what IO-figures do you get on the other Linux CI-servers?

[josteink]

@mmitche That's odd. Building something like coreclr is fairly IO-intensive so I did some basic IO-testing on our Interroute VMs vs our more limited Azure VMs.

Although there's probably a million errors in the measuremeant technique scientifically speaking, the difference is night and day:

# Interroute
$ dd if=/dev/zero of=test bs=1M count=1000 
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 0.624000 secs (1,680,410,061 bytes/sec)

# Azure
$  dd if=/dev/zero of=test bs=1M count=1000 
1000+0 records in
1000+0 records out
1048576000 bytes transferred in 130.005050 secs (8,065,656 bytes/sec)

What causes this massive difference in disk-IO, I don't know. It might be available memory available for caching. It might be the ZFS filesystem configuration (a good volume for the ZIL?). The fact of the matter is that we're getting around 250x the IO throughput on Interroute's VMware-based servers.

If we can boost the IO-throughput on our Azure-machines we should be able to bring down the build-time significantly. For a CI-server you probably want more than 8MB/s. Out of interest, what IO-figures do you get on the other Linux CI-servers?

[wizardbeard]

Congratulations!!! This is exciting news! FreeBSD is my favorite OS and .NET/C# is my favorite development tooling. This makes me rather happy.

[wizardbeard]

Congratulations!!! This is exciting news! FreeBSD is my favorite OS and .NET/C# is my favorite development tooling. This makes me rather happy.

[josteink]

I guess we do have a CI-build now, so this issue should be considered closed. I'll take potential issues with CI-build to https://github.com/dotnet/dotnet-ci .

[josteink]

I guess we do have a CI-build now, so this issue should be considered closed. I'll take potential issues with CI-build to https://github.com/dotnet/dotnet-ci .