Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Long-running mono process segfaults #1697

Closed
rudism opened this issue Jul 20, 2015 · 2 comments
Closed

Long-running mono process segfaults #1697

rudism opened this issue Jul 20, 2015 · 2 comments
Labels
bug

Comments

@rudism
Copy link

@rudism rudism commented Jul 20, 2015

I am having an issue with a shippable build that installs mono, compiles a project, and then runs some integration tests.

Here is an example of a failed build:
https://app.shippable.com/builds/55ad582a696c7f0b00588662

The step that fails is using mono to run the xunit console runner against our integration tests DLL. The integration tests are probably the most resource intensive step of our build--it downloads and restores a postgresql database from S3, then spins up local copies of our application (which get linked to a copy of the restored database, and tested by making local web requests against). We have a number of different test suites that run which each get their own copy of the database (to prevent each suite from affecting the data that the others have access to). This takes quite a long time to complete, and has successfully completed in some builds, but more often than not I get a segmentation fault in the mono executable.

When I load up ubuntu1404_base into a local docker container on my machine and run all of the commands from my shippable yml file locally, everything works fine every time (no seg faults), which leads me to believe there is some difference or perhaps some limitation being imposed on the process when Shippable runs it, but I am at a loss with trying to figure out how to debug or track this problem down.

Any help, suggestions, or guidance would be appreciated.

@rudism

This comment has been minimized.

Copy link
Author

@rudism rudism commented Aug 26, 2015

Despite several different approaches and attempts, we have been unable to get mono working at all on Shippable. The following scenarios all result in random segfaults while executing mono processes:

  • apt-get installed mono on shippableimages/ubuntu1404_base
  • mono built from source on shippableimages/ubuntu1404_base and on ubuntu:14.04
  • mono built from source on centos:6

The centos:6 image was rock-solid when running mono locally (not a single segfault after hours of testing long-running build processes), but still had random segfaults during almost every build when run on Shippable.

We also tried a centos:7 based image, but it would hang on Shippable every time before even starting the build.

The only way we have been able to work around this issue is to offload the build process into another externally hosted container. Our commands in the shippable.yml file for our mono project(s) now look like this:

  1. Spin up a Centos7 container on Amazon's ECS service using the AWS CLI
  2. SSH commands to the remote container to build the project and run our tests
  3. SCP the test results back to the Shippable container

Obviously this is highly inefficient and sub-optimal, but without being able to run mono reliably directly on the Shippable containers it seems to be our only option. We would still prefer to avoid having to run our own external build servers in AWS if Shippable can fix the problems with mono.

@manishas

This comment has been minimized.

Copy link
Contributor

@manishas manishas commented Mar 8, 2016

This is an old issue opened before our build system upgrade and build image refresh. Closing since it is connected to old build platform.

@manishas manishas closed this Mar 8, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.