Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Segfaults #369

Open
mainameiz opened this issue Oct 15, 2018 · 12 comments
Open

Segfaults #369

mainameiz opened this issue Oct 15, 2018 · 12 comments
Assignees

Comments

@mainameiz
Copy link

mainameiz commented Oct 15, 2018

Sometimes cc-test-reporter fails with segfaults.

  • It segfaults on before-build, sum-coverage, upload-coverage commands
  • We use docker-compose to run our tests
  • I have tried to wrap cc-test-reporter into "retry loop" but it does not work - it segfaults every time.
cc-test-reporter() {
  set +e # temporary disable "fail fast" mode

  try_num=0
  while true; do
    ./cc-test-reporter "$@"
    code="$?"
    if [ $code -eq 0 ] ; then
      break
    else
      echo "cc-test-reporter failed. Code: ${code}."
      try_num=$(($try_num + 5))
      if [ $try_num -lt 5 ] ; then
        echo "Retry #$try_num..."
      else
        echo "Retries exceeded..."
        echo "Sending core dump to ftp"
        ls -lah ./
        sftp ${SFTP_USER}@${SFTP_HOST}:/sftp <<< "
          -mkdir publisher
          cd publisher
          put core
          exit
        "
        exit "$code"
      fi
    fi
  done

  set -e
}

How can I help you to investigate this issue?

@mainameiz
Copy link
Author

I can send you core dump or rebuild cc-test-reporter with some debug flags...

@wfleming
Copy link
Contributor

I can send you core dump or rebuild cc-test-reporter with some debug flags...

Both would be useful @mainameiz, thanks.

Can you also please confirm what URL you're installing the test reporter binary from? And provide some details about the environment you're running this in? (OS version, what CI system, etc.). Thanks.

@mainameiz
Copy link
Author

I download fresh version of cc-test-reporter for each build from https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64.

Which ldflags should I use?

@wfleming
Copy link
Contributor

Let's hold off on having you compile with custom ldflags until we've eliminated a few other possibilities, I'm not sure offhand what extra ldflags would be useful.

What are the details of the environment you're running the test-reporter in? E.g. OS version, CI system, what hardware it's running on if you know. Output of uname -a might be helpful.

If you don't want to post the coredump here, you can email it to me: will [at] codeclimate.com.

@mainameiz
Copy link
Author

We use

  • CentOS 7
    $ uname -a
    Linux somehost.com 4.14.62-6.el7.x86_64 #1 SMP Thu Aug 9 20:58:00 IST 2018 x86_64 x86_64 x86_64 GNU/Linux
    $ cat /etc/centos-release
    CentOS Linux release 7.5.1804 (Core)
    
  • Jenkins 1.9.0 Core 2.146 7b2cf06 10th October 2018 10:29 PM

We also used https://ci.vexor.io some time ago, and had the same problems. It was Ubuntu (probably 16.04).

I will send you coredump on email next time it will crash.

@mainameiz
Copy link
Author

About hardware. It reproduces on virtual machine, on bare metal instance and in another CI (as I wrote above about https://ci.vexor.io)

@mainameiz
Copy link
Author

Sent you a coredump at email.

@mainameiz
Copy link
Author

Today we faced a huge amount of segfaults. Every CI build fails.
We also compile cc-test-reporter with debug flags (https://gist.github.com/dmexe/f3309c470b30df0f85695675559ffde7) and it seems it works without segfaults.

@wfleming
Copy link
Contributor

Thanks @mainameiz.

Unfortunately I haven't been able to extract anything useful from the coredump using dlv. I should have thought of this before, but with a release binary there won't be any debug symbols, and there are known limitations using delve with release binaries AFAICT.

The fact that a binary you've compiled yourself is working fine is instructive. I have suspected that there is probably something unusual with your CI environment causing an incompatibility with our release binaries, and so recompiling your own binary in a similar environment is likely to fix it. I suspect it would still work for you if you compiled your own binary even without debug flags, but it's probably a good idea to keep them on for so that if this new binary eventually segfaults the coredumps will be more readable. Hopefully the custom binary is stable for you, but if it isn't please share those details with us.

@mainameiz
Copy link
Author

@wfleming Thanks for a quick feedback.

there is probably something unusual with your CI environment

I wish to draw attention that this problem occurred in different environments. In our case it is reproducable in Jenkins+CentOS and vexor.io (they have some sort of Ubuntu)

We will continue to use our own build of cc-test-reporter and I will write if the problem appears again.

@mainameiz
Copy link
Author

Hello, Will.

Our version of cc-test-reporter also segfaults. Just yesterday. It was the first time after the moment we start to use our own version. It segfaults on before-build command. Before that moment rerun of tests of the same commit resolves the issue, but now it reproduces on the same commit again and again.
Coredump now contains debug symbols (https://gist.github.com/dmexe/f3309c470b30df0f85695675559ffde7), hope this weill help you to investigate the problem.
I send coredump and cc-test-reporter binary to your email.

@mainameiz
Copy link
Author

Is there any news?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants