Segfaults

[mainameiz]

Sometimes cc-test-reporter fails with segfaults.

• It segfaults on before-build, sum-coverage, upload-coverage commands
• We use docker-compose to run our tests
• I have tried to wrap cc-test-reporter into "retry loop" but it does not work - it segfaults every time.

cc-test-reporter() {
  set +e # temporary disable "fail fast" mode

  try_num=0
  while true; do
    ./cc-test-reporter "$@"
    code="$?"
    if [ $code -eq 0 ] ; then
      break
    else
      echo "cc-test-reporter failed. Code: ${code}."
      try_num=$(($try_num + 5))
      if [ $try_num -lt 5 ] ; then
        echo "Retry #$try_num..."
      else
        echo "Retries exceeded..."
        echo "Sending core dump to ftp"
        ls -lah ./
        sftp ${SFTP_USER}@${SFTP_HOST}:/sftp <<< "
          -mkdir publisher
          cd publisher
          put core
          exit
        "
        exit "$code"
      fi
    fi
  done

  set -e
}

How can I help you to investigate this issue?

[mainameiz]

I can send you core dump or rebuild cc-test-reporter with some debug flags...

[wfleming]

I can send you core dump or rebuild cc-test-reporter with some debug flags...

Both would be useful @mainameiz, thanks.

Can you also please confirm what URL you're installing the test reporter binary from? And provide some details about the environment you're running this in? (OS version, what CI system, etc.). Thanks.

[mainameiz]

I download fresh version of cc-test-reporter for each build from https://codeclimate.com/downloads/test-reporter/test-reporter-latest-linux-amd64.

Which ldflags should I use?

[wfleming]

Let's hold off on having you compile with custom ldflags until we've eliminated a few other possibilities, I'm not sure offhand what extra ldflags would be useful.

What are the details of the environment you're running the test-reporter in? E.g. OS version, CI system, what hardware it's running on if you know. Output of uname -a might be helpful.

If you don't want to post the coredump here, you can email it to me: will [at] codeclimate.com.

[mainameiz]

We use

• CentOS 7

  $ uname -a
  Linux somehost.com 4.14.62-6.el7.x86_64 #1 SMP Thu Aug 9 20:58:00 IST 2018 x86_64 x86_64 x86_64 GNU/Linux
  $ cat /etc/centos-release
  CentOS Linux release 7.5.1804 (Core)
  

• Jenkins 1.9.0 Core 2.146 7b2cf06 10th October 2018 10:29 PM

We also used https://ci.vexor.io some time ago, and had the same problems. It was Ubuntu (probably 16.04).

I will send you coredump on email next time it will crash.

[mainameiz]

About hardware. It reproduces on virtual machine, on bare metal instance and in another CI (as I wrote above about https://ci.vexor.io)

[mainameiz]

Sent you a coredump at email.

[mainameiz]

Today we faced a huge amount of segfaults. Every CI build fails.
We also compile cc-test-reporter with debug flags (https://gist.github.com/dmexe/f3309c470b30df0f85695675559ffde7) and it seems it works without segfaults.

[wfleming]

Thanks @mainameiz.

Unfortunately I haven't been able to extract anything useful from the coredump using dlv. I should have thought of this before, but with a release binary there won't be any debug symbols, and there are known limitations using delve with release binaries AFAICT.

The fact that a binary you've compiled yourself is working fine is instructive. I have suspected that there is probably something unusual with your CI environment causing an incompatibility with our release binaries, and so recompiling your own binary in a similar environment is likely to fix it. I suspect it would still work for you if you compiled your own binary even without debug flags, but it's probably a good idea to keep them on for so that if this new binary eventually segfaults the coredumps will be more readable. Hopefully the custom binary is stable for you, but if it isn't please share those details with us.

[mainameiz]

@wfleming Thanks for a quick feedback.

there is probably something unusual with your CI environment

I wish to draw attention that this problem occurred in different environments. In our case it is reproducable in Jenkins+CentOS and vexor.io (they have some sort of Ubuntu)

We will continue to use our own build of cc-test-reporter and I will write if the problem appears again.

[mainameiz]

Hello, Will.

Our version of cc-test-reporter also segfaults. Just yesterday. It was the first time after the moment we start to use our own version. It segfaults on before-build command. Before that moment rerun of tests of the same commit resolves the issue, but now it reproduces on the same commit again and again.
Coredump now contains debug symbols (https://gist.github.com/dmexe/f3309c470b30df0f85695675559ffde7), hope this weill help you to investigate the problem.
I send coredump and cc-test-reporter binary to your email.

[mainameiz]

Is there any news?