New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

YML - How? #3987

Closed
UdiSemana opened this Issue Nov 30, 2017 · 10 comments

Comments

Projects
None yet
6 participants
@UdiSemana

UdiSemana commented Nov 30, 2017

Description of your issue:

https://i.imgur.com/ep3P2hk.png
NOTE: If you are experiencing a build failure, please include:

  • Link to the build failure
  • Scripts called from the yml (optional)
@trriplejay

This comment has been minimized.

Show comment
Hide comment
@trriplejay

trriplejay Nov 30, 2017

It looks like you need a shippable.yml file at the root of your repository. See our docs here for information on what that file can contain: http://docs.shippable.com/ci/yml-structure/

trriplejay commented Nov 30, 2017

It looks like you need a shippable.yml file at the root of your repository. See our docs here for information on what that file can contain: http://docs.shippable.com/ci/yml-structure/

@UdiSemana

This comment has been minimized.

Show comment
Hide comment
@UdiSemana

UdiSemana Dec 3, 2017

UdiSemana commented Dec 3, 2017

@a-murphy

This comment has been minimized.

Show comment
Hide comment
@a-murphy

a-murphy Dec 4, 2017

Although we don't have any default commands for C# projects, you can still use Shippable with C# projects.

One way to do this is by specifying language: none and the image you would like to use in pre_ci_boot in your shippable.yml as documented here. Examples of pre_ci_boot are available in this documentation.

Or, if your project has many of the same dependencies as typical projects in a supported language, you can specify that language in your shippable.yml instead of C#. The language in shippable.yml is not required to match the project language; it's used to determine the default image, default commands, and where it will look for the language version it will use, but commands in the ci section will override the default commands.

a-murphy commented Dec 4, 2017

Although we don't have any default commands for C# projects, you can still use Shippable with C# projects.

One way to do this is by specifying language: none and the image you would like to use in pre_ci_boot in your shippable.yml as documented here. Examples of pre_ci_boot are available in this documentation.

Or, if your project has many of the same dependencies as typical projects in a supported language, you can specify that language in your shippable.yml instead of C#. The language in shippable.yml is not required to match the project language; it's used to determine the default image, default commands, and where it will look for the language version it will use, but commands in the ci section will override the default commands.

@ambarish2012

This comment has been minimized.

Show comment
Hide comment
@ambarish2012

ambarish2012 Dec 4, 2017

Contributor

@UdiSemana in addition to the excellent suggestions by @a-murphy , please note that we do not support windows containers yet. So you will have to use Monodevelop to compile your C# project in a Linux container. The list of supported Linux distributions for Mono can be found here. You can build a custom image with any of these distributions with the Monodevelop toolset.

Contributor

ambarish2012 commented Dec 4, 2017

@UdiSemana in addition to the excellent suggestions by @a-murphy , please note that we do not support windows containers yet. So you will have to use Monodevelop to compile your C# project in a Linux container. The list of supported Linux distributions for Mono can be found here. You can build a custom image with any of these distributions with the Monodevelop toolset.

@seniorquico

This comment has been minimized.

Show comment
Hide comment
@seniorquico

seniorquico Dec 4, 2017

@UdiSemana I can confirm that some assembly is required when building C# projects on Shippable. However, it totally works. I've used two different approaches, and they've worked out great for our different projects. Of note, we have successfully built and run production loads on Windows boxes using binaries built using Shippable's Linux nodes!

First approach: I assembled an Docker image based on the dry-dock/u16 base that includes Mono 5. Mono 5 uses the same MSBuild task runner & Roslyn compiler as VS 2017. This is the build environment we use to produce binaries that we can run on our Windows clients. This may not work if you have Windows-only tools in your build, though. I'll try to post the Dockerfile on GitHub for reference.

https://hub.docker.com/r/seniorquico/u16dotnet/

Second approach: A number of our servers are now running C# web applications and C# backend services that run on Linux (on K8s clusters) using the .NET Core CLR. For these servers, we assemble Docker images using docker-compose and Microsoft's official build images.

https://hub.docker.com/r/microsoft/dotnet/
https://hub.docker.com/r/microsoft/aspnetcore-build/

I was hoping to get an opportunity to blog about our Shippable setup with C#-based apps; however, I'm in a big crunch at work. I may be able to contribute more to this thread if you have questions, though.

seniorquico commented Dec 4, 2017

@UdiSemana I can confirm that some assembly is required when building C# projects on Shippable. However, it totally works. I've used two different approaches, and they've worked out great for our different projects. Of note, we have successfully built and run production loads on Windows boxes using binaries built using Shippable's Linux nodes!

First approach: I assembled an Docker image based on the dry-dock/u16 base that includes Mono 5. Mono 5 uses the same MSBuild task runner & Roslyn compiler as VS 2017. This is the build environment we use to produce binaries that we can run on our Windows clients. This may not work if you have Windows-only tools in your build, though. I'll try to post the Dockerfile on GitHub for reference.

https://hub.docker.com/r/seniorquico/u16dotnet/

Second approach: A number of our servers are now running C# web applications and C# backend services that run on Linux (on K8s clusters) using the .NET Core CLR. For these servers, we assemble Docker images using docker-compose and Microsoft's official build images.

https://hub.docker.com/r/microsoft/dotnet/
https://hub.docker.com/r/microsoft/aspnetcore-build/

I was hoping to get an opportunity to blog about our Shippable setup with C#-based apps; however, I'm in a big crunch at work. I may be able to contribute more to this thread if you have questions, though.

@ambarish2012

This comment has been minimized.

Show comment
Hide comment
@ambarish2012

ambarish2012 Dec 4, 2017

Contributor

@seniorquico thrilled to hear about the success you have had building C# binaries using mono on Linux. I could provide writing support for this awesome blog if you are interested. It would help a ton of customers jump start their efforts on getting CI and potentially CD (to K8s clusters) working for their C# projects.

Contributor

ambarish2012 commented Dec 4, 2017

@seniorquico thrilled to hear about the success you have had building C# binaries using mono on Linux. I could provide writing support for this awesome blog if you are interested. It would help a ton of customers jump start their efforts on getting CI and potentially CD (to K8s clusters) working for their C# projects.

@seniorquico

This comment has been minimized.

Show comment
Hide comment
@seniorquico

seniorquico Dec 5, 2017

@ambarish2012 All right... you sucked me in. 😄 Here are my raw notes. Let me know if you do end up posting it somewhere. I'd love to give it a review. I'd also love to hear if you have any interesting insights or input on these approaches. It's definitely been an interesting, fun journey using Shippable's Linux nodes to build our C# applications.

For the first approach, here's a repo with the Dockerfile and install script I used to create the build image:

https://github.com/seniorquico/shippable-u16dotnet

Reviewing the Dockerfile, I remembered that I had encountered errors getting the VS 2017 projects to build using Mono 5.0 (the latest, stable package version at the time). It was resolved by using the latest beta package version, 5.2, at the time. It appears Mono 5.4 is released as stable, so my use of the beta package is (hopefully) unnecessary today.

https://stackoverflow.com/questions/45015183/compiling-c-sharp-7-code-containing-valuetuple-with-mono-5/45026409

Here are the relevant bits of a Shippable CI job using this image:

language: none

build:
  cache: true
  cache_dir_list:
    - /root/.config/NuGet
    - /root/.nuget
  pre_ci_boot:
    image_name: seniorquico/u16dotnet
    image_tag: "v5.7.1-5.2.0.196-1"
    options: "-e HOME=/root"
  ci:
    - mono --version
    - msbuild /version
    - csc /version
    - echo 'Restoring NuGet dependencies...'
    - msbuild /target:restore /property:Configuration=Release /property:Platform="Any CPU" /maxcpucount /toolsversion:15.0 /verbosity:detailed Project.sln
    - echo 'Building project...'
    - msbuild /property:Configuration=Release /property:Platform="Any CPU" /maxcpucount /toolsversion:15.0 /verbosity:detailed Project.sln

For our particular project, we were successful using the above to build Windows-based services (pushing the artifacts to Azure Blob storage that kicked off a non-Shippable CD process) and an Azure App Service web site (pushing the artifacts directly to the App Service using Kudu). This approach should work if you're looking for a full .NET Framework compatible build.

Quick detour-- When starting in on the second approach, we first tried using the Docker builder pattern and the previously mentioned Microsoft build images. The default ASP.NET Core project template-- created using the dotnet CLI-- produced a Dockerfile like the following:

FROM microsoft/aspnetcore:2.0 AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/aspnetcore-build:2.0 AS build
WORKDIR /src
COPY *.sln ./
COPY src/
COPY test/
RUN dotnet restore
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Project.dll"]

While this worked just fine, we found it left something to be desired wrt VS 2017 tooling, integration, and debugging in a "real" container.

For our second approach, we decided to jump onto the VS 2017 extension Visual Studio Tools for Docker to address the above concerns. The extension provides a docker-compose project, integration with Docker for Windows, and a native debugging solution. It ends up using the same Microsoft build images with a few tweaks (mainly the removal of the builder pattern).

Of note, we had to switch to the "microsoft/aspnetcore-build:1.0-2.0-stretch" build image. This particular build comes bundled with the needed MSBuild tasks for the docker-compose project. Using the original build images will result in an error indicated the docker-compose project is not a recognized project type.

https://stackoverflow.com/questions/45695089/how-to-build-push-container-from-docker-compose-ci-build-yml-file/47605919

The docker-compose project generated by the extension comes with a "ci build" template. I simply extended that with a Shippable-specific override. My "docker-compose.ci.build.shippable.yml" contains:

version: '3'

services:
  ci-build:
    volumes:
      - /code:/src

http://docs.shippable.com/ci/docker-compose/
#2938

Otherwise, the other docker-compose YML files are the same as produced by the VS 2017 tooling. We ran the build something like:

language: none

build:
  pre_ci_boot:
    options: "-v /code:/code"
  ci:
    - echo 'Installing Docker Compose...'
    - shipctl retry curl -sSL "https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-$( uname -s )-$( uname -m )" -o /usr/local/bin/docker-compose
    - chmod +x /usr/local/bin/docker-compose
    - docker-compose --version
    - echo 'Copying source code to Docker volume...'
    - cd "${SHIPPABLE_BUILD_DIR}/project"
    - cp -R . /code
    - echo 'Building libraries and applications...'
    - cd /code
    - docker-compose -f ./docker-compose.ci.build.yml -f ./docker-compose.ci.build.shippable.yml up
    - echo 'Building Docker images...'
    - docker-compose -f ./docker-compose.yml build
    - echo 'Tagging Docker images...'
    - docker tag myproject.azurecr.io/service/api "myproject.azurecr.io/service/api:${COMMIT}"
    - docker images
    - echo 'Cleaning up environment...'
    - docker-compose -f ./docker-compose.ci.build.yml -f ./docker-compose.ci.build.shippable.yml down

For our particular project, we are currently deploying builds from the above to a staging K8s environment running on the new Azure Container Service (AKS). (Another side note... we also had to manually upgrade the Azure CLI in order to work with AKS.) This approach should work if you're looking for .NET Core CLR capable builds and/or Docker images.

Apologies if any of the snippets above have errors. I had to remove quite a bit to get to the core of what was going on.

In summary: Mono 5 worked great for building applications that needed to target the full .NET Framework. The Docker builder pattern demonstrated by the ASP.NET Core project template worked great to build images one at a time targeting the .NET Core CLR. The Docker Compose pattern demonstrated by the Visual Studio Tools for Docker also worked great to build images targeting the .NET Core CLR. The extension also provided our team with a great developer workflow.

seniorquico commented Dec 5, 2017

@ambarish2012 All right... you sucked me in. 😄 Here are my raw notes. Let me know if you do end up posting it somewhere. I'd love to give it a review. I'd also love to hear if you have any interesting insights or input on these approaches. It's definitely been an interesting, fun journey using Shippable's Linux nodes to build our C# applications.

For the first approach, here's a repo with the Dockerfile and install script I used to create the build image:

https://github.com/seniorquico/shippable-u16dotnet

Reviewing the Dockerfile, I remembered that I had encountered errors getting the VS 2017 projects to build using Mono 5.0 (the latest, stable package version at the time). It was resolved by using the latest beta package version, 5.2, at the time. It appears Mono 5.4 is released as stable, so my use of the beta package is (hopefully) unnecessary today.

https://stackoverflow.com/questions/45015183/compiling-c-sharp-7-code-containing-valuetuple-with-mono-5/45026409

Here are the relevant bits of a Shippable CI job using this image:

language: none

build:
  cache: true
  cache_dir_list:
    - /root/.config/NuGet
    - /root/.nuget
  pre_ci_boot:
    image_name: seniorquico/u16dotnet
    image_tag: "v5.7.1-5.2.0.196-1"
    options: "-e HOME=/root"
  ci:
    - mono --version
    - msbuild /version
    - csc /version
    - echo 'Restoring NuGet dependencies...'
    - msbuild /target:restore /property:Configuration=Release /property:Platform="Any CPU" /maxcpucount /toolsversion:15.0 /verbosity:detailed Project.sln
    - echo 'Building project...'
    - msbuild /property:Configuration=Release /property:Platform="Any CPU" /maxcpucount /toolsversion:15.0 /verbosity:detailed Project.sln

For our particular project, we were successful using the above to build Windows-based services (pushing the artifacts to Azure Blob storage that kicked off a non-Shippable CD process) and an Azure App Service web site (pushing the artifacts directly to the App Service using Kudu). This approach should work if you're looking for a full .NET Framework compatible build.

Quick detour-- When starting in on the second approach, we first tried using the Docker builder pattern and the previously mentioned Microsoft build images. The default ASP.NET Core project template-- created using the dotnet CLI-- produced a Dockerfile like the following:

FROM microsoft/aspnetcore:2.0 AS base
WORKDIR /app
EXPOSE 80

FROM microsoft/aspnetcore-build:2.0 AS build
WORKDIR /src
COPY *.sln ./
COPY src/
COPY test/
RUN dotnet restore
RUN dotnet build -c Release -o /app

FROM build AS publish
RUN dotnet publish -c Release -o /app

FROM base AS final
WORKDIR /app
COPY --from=publish /app .
ENTRYPOINT ["dotnet", "Project.dll"]

While this worked just fine, we found it left something to be desired wrt VS 2017 tooling, integration, and debugging in a "real" container.

For our second approach, we decided to jump onto the VS 2017 extension Visual Studio Tools for Docker to address the above concerns. The extension provides a docker-compose project, integration with Docker for Windows, and a native debugging solution. It ends up using the same Microsoft build images with a few tweaks (mainly the removal of the builder pattern).

Of note, we had to switch to the "microsoft/aspnetcore-build:1.0-2.0-stretch" build image. This particular build comes bundled with the needed MSBuild tasks for the docker-compose project. Using the original build images will result in an error indicated the docker-compose project is not a recognized project type.

https://stackoverflow.com/questions/45695089/how-to-build-push-container-from-docker-compose-ci-build-yml-file/47605919

The docker-compose project generated by the extension comes with a "ci build" template. I simply extended that with a Shippable-specific override. My "docker-compose.ci.build.shippable.yml" contains:

version: '3'

services:
  ci-build:
    volumes:
      - /code:/src

http://docs.shippable.com/ci/docker-compose/
#2938

Otherwise, the other docker-compose YML files are the same as produced by the VS 2017 tooling. We ran the build something like:

language: none

build:
  pre_ci_boot:
    options: "-v /code:/code"
  ci:
    - echo 'Installing Docker Compose...'
    - shipctl retry curl -sSL "https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-$( uname -s )-$( uname -m )" -o /usr/local/bin/docker-compose
    - chmod +x /usr/local/bin/docker-compose
    - docker-compose --version
    - echo 'Copying source code to Docker volume...'
    - cd "${SHIPPABLE_BUILD_DIR}/project"
    - cp -R . /code
    - echo 'Building libraries and applications...'
    - cd /code
    - docker-compose -f ./docker-compose.ci.build.yml -f ./docker-compose.ci.build.shippable.yml up
    - echo 'Building Docker images...'
    - docker-compose -f ./docker-compose.yml build
    - echo 'Tagging Docker images...'
    - docker tag myproject.azurecr.io/service/api "myproject.azurecr.io/service/api:${COMMIT}"
    - docker images
    - echo 'Cleaning up environment...'
    - docker-compose -f ./docker-compose.ci.build.yml -f ./docker-compose.ci.build.shippable.yml down

For our particular project, we are currently deploying builds from the above to a staging K8s environment running on the new Azure Container Service (AKS). (Another side note... we also had to manually upgrade the Azure CLI in order to work with AKS.) This approach should work if you're looking for .NET Core CLR capable builds and/or Docker images.

Apologies if any of the snippets above have errors. I had to remove quite a bit to get to the core of what was going on.

In summary: Mono 5 worked great for building applications that needed to target the full .NET Framework. The Docker builder pattern demonstrated by the ASP.NET Core project template worked great to build images one at a time targeting the .NET Core CLR. The Docker Compose pattern demonstrated by the Visual Studio Tools for Docker also worked great to build images targeting the .NET Core CLR. The extension also provided our team with a great developer workflow.

@ambarish2012

This comment has been minimized.

Show comment
Hide comment
@ambarish2012

ambarish2012 Dec 5, 2017

Contributor

@seniorquico thanks for the great notes. I am thinking of splitting this into two blogs. The first one being a simple CI blog for building a C# application using NuGet packages using a custom image based of u16 that has mono 5.4 pre-installed using your excellent approach. I will leverage your custom image in the blog.

We are going to release a managed deploy job next week that will allow deploying images to AKS. I would like to combine your docker-compose based CI with our AKS CD for a second blog.

Please could you send me your email address so that we can discuss details offline.

Contributor

ambarish2012 commented Dec 5, 2017

@seniorquico thanks for the great notes. I am thinking of splitting this into two blogs. The first one being a simple CI blog for building a C# application using NuGet packages using a custom image based of u16 that has mono 5.4 pre-installed using your excellent approach. I will leverage your custom image in the blog.

We are going to release a managed deploy job next week that will allow deploying images to AKS. I would like to combine your docker-compose based CI with our AKS CD for a second blog.

Please could you send me your email address so that we can discuss details offline.

@seniorquico

This comment has been minimized.

Show comment
Hide comment
@seniorquico

seniorquico Dec 5, 2017

@ambarish2012 That sounds great. You can reach me at kyledodson@gmail.com.

seniorquico commented Dec 5, 2017

@ambarish2012 That sounds great. You can reach me at kyledodson@gmail.com.

@seniorquico

This comment has been minimized.

Show comment
Hide comment
@seniorquico

seniorquico Dec 6, 2017

For anyone that may be following along and is interested in building C# projects that target the .NET Framework (as opposed to the .NET Core CLR)...

Here's an example project and the logs for a Shippable CI run that built both a "classic" csproj-based solution (primarily meaning a packages.config file and a packages folder in the solution directory) and a "modern" csproj-based solution (the Visual Studio 2017 format that puts NuGet references in the csproj file). Both of these example projects target the .NET Framework 4.6.1.

https://app.shippable.com/github/seniorquico/shippable-u16dotnet-example/runs/2/1/console
https://github.com/seniorquico/shippable-u16dotnet-example

I also pushed an updated image to Docker Hub based on the v5.10.4 Shippable image that includes stable Mono 5.4.1.6.

seniorquico commented Dec 6, 2017

For anyone that may be following along and is interested in building C# projects that target the .NET Framework (as opposed to the .NET Core CLR)...

Here's an example project and the logs for a Shippable CI run that built both a "classic" csproj-based solution (primarily meaning a packages.config file and a packages folder in the solution directory) and a "modern" csproj-based solution (the Visual Studio 2017 format that puts NuGet references in the csproj file). Both of these example projects target the .NET Framework 4.6.1.

https://app.shippable.com/github/seniorquico/shippable-u16dotnet-example/runs/2/1/console
https://github.com/seniorquico/shippable-u16dotnet-example

I also pushed an updated image to Docker Hub based on the v5.10.4 Shippable image that includes stable Mono 5.4.1.6.

@manishas manishas closed this Dec 19, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment