[ci] dowgrade gcc version for Linux on Azure #1718

StrikerRUS · 2018-09-30T17:40:59Z

Refer to #1708.

StrikerRUS · 2018-09-30T17:51:41Z

GPU test fails. I suppose that the reason is in these lines:

2018-09-30T17:18:46.9243394Z Preparing to unpack .../ocl-icd-libopencl1_2.2.8-1_amd64.deb ...
2018-09-30T17:18:46.9646898Z Unpacking ocl-icd-libopencl1:amd64 (2.2.8-1) ...
2018-09-30T17:18:47.1272916Z Selecting previously unselected package opencl-headers.
2018-09-30T17:18:47.1380157Z Preparing to unpack .../opencl-headers_2.0~svn32091-2_all.deb ...
2018-09-30T17:18:47.1548332Z Unpacking opencl-headers (2.0~svn32091-2) ...
2018-09-30T17:18:47.2831063Z Selecting previously unselected package ocl-icd-opencl-dev:amd64.
2018-09-30T17:18:47.2923796Z Preparing to unpack .../ocl-icd-opencl-dev_2.2.8-1_amd64.deb ...
2018-09-30T17:18:47.3099881Z Unpacking ocl-icd-opencl-dev:amd64 (2.2.8-1) ...
2018-09-30T17:18:47.3866224Z Processing triggers for libc-bin (2.23-0ubuntu10) ...
2018-09-30T17:18:47.5810345Z Setting up ocl-icd-libopencl1:amd64 (2.2.8-1) ...
2018-09-30T17:18:47.6531984Z Setting up opencl-headers (2.0~svn32091-2) ...
2018-09-30T17:18:47.7040672Z Setting up ocl-icd-opencl-dev:amd64 (2.2.8-1) ...
---> 2018-09-30T17:18:47.7545016Z Processing triggers for libc-bin (2.23-0ubuntu10) ... <---

While installing OpenCL libraries libc is being updated and becomes incompatible with g++-4.8. It's my assumption.

@guolinke Since we don't provide any artifacts from GPU test publicly, what do you think about the following?

elif [[ $AZURE == "true" ]] && [[ $COMPILER == "gcc" ]] && [[ $TASK != "GPU" ]]; then
  <downgrade gcc>

guolinke · 2018-10-01T01:55:45Z

@StrikerRUS yeah, that is okay

guolinke · 2018-10-01T02:08:59Z

.ci/setup.sh

-    if [[ $AZURE == "true" ]] && [[ $COMPILER == "clang" ]]; then
-        update-alternatives --install /usr/bin/clang++ clang++ /usr/bin/clang++-6.0 100
-        update-alternatives --install /usr/bin/clang clang /usr/bin/clang-6.0 100
+    if [[ $AZURE == "true" ]]; then
        sudo apt-get update


may be we should update for travis as well, since line 48 will install ocl-icd-opencl-dev

I tested without sudo apt-get update at Travis - worked well. But if it doesn't significantly enlarge the test time, lets leave this.

guolinke · 2018-10-01T02:32:01Z

@StrikerRUS
It seems the build binaries still require glibc 2.23:

guoke@test-vm3:~$ ldd lib_lightgbm.so
./lib_lightgbm.so: /lib/x86_64-linux-gnu/libm.so.6: version `GLIBC_2.23' not found (required by ./lib_lightgbm.so)
        linux-vdso.so.1 =>  (0x00007ffc472ae000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f8ecbcc6000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f8ecb9c0000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f8ecb7b1000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f8ecb59b000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f8ecb37d000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f8ecafb4000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f8ecc445000)

So i add some fixes .

guolinke · 2018-10-01T04:20:41Z

include/LightGBM/meta.h

@@ -62,6 +62,10 @@ typedef void(*AllgatherFunction)(char* input, comm_size_t input_size, const comm
 #define __func__ __FUNCTION__
 #endif

+#ifdef GCC
+__asm__(".symver lgamma, lgamma@GLIBC_2.2.5");


maybe a better solution is implementing a lgamma by ourself.

Hmm, why is this symver needed?

lgamma will introduce the glibc2.23 dependency.

This sounds a bit odd. I've checked the glibc changelog and it seems lgamma have been there for quite a while.

I think lgamma have a new implementation in glibc2.23. and result in this new dependency.

StrikerRUS · 2018-10-01T11:19:59Z

CMakeLists.txt

@@ -17,6 +17,10 @@ if(APPLE)
    OPTION(APPLE_OUTPUT_DYLIB "Output dylib shared library" OFF)
 endif()

+if((CMAKE_CXX_COMPILER_ID STREQUAL "GNU") AND (UNIX) AND (NOT APPLE))
+  ADD_DEFINITIONS(-DGCC)
+endif()


Maybe it's better to move this somewhere lower to other definitions?

sorry, I didn't understand it, can you describe more details ?

I mean, move these lines lower in the file. For example, somewhere around here
https://github.com/Microsoft/LightGBM/blob/606eeefeb8e1f217a0f91fa0f11137125942b549/CMakeLists.txt#L66-L68
or here
https://github.com/Microsoft/LightGBM/blob/606eeefeb8e1f217a0f91fa0f11137125942b549/CMakeLists.txt#L118-L119
where other ADD_DEFINITIONS are set.

guolinke · 2018-10-01T14:42:15Z

@StrikerRUS what do you think about the current fix for lgamma ?

StrikerRUS · 2018-10-01T16:47:52Z

@guolinke To be honest, I'm not familiar with SymbolVersioning, but I think it's rather hacky solution and intuitively I can't say that I like it. I'm afraid that it will cause some troubles.

I'd better prefer this your proposal:

maybe a better solution is implementing a lgamma by ourself.

Can we just copy-paste this function from the sources?

guolinke · 2018-10-02T01:33:00Z

@StrikerRUS I just notice we don't need the lgamma. The new fix is pushed.

guoke@test-vm3:~$ ldd lib_lightgbm.so
        linux-vdso.so.1 =>  (0x00007ffdfc5e5000)
        libstdc++.so.6 => /usr/lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f1bc6a8e000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f1bc6788000)
        libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f1bc6579000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f1bc6363000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f1bc6145000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f1bc5d7c000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f1bc720d000)

superbobry · 2018-10-02T09:20:53Z

What is the glibc version used by GCC 4.8? Could you mention it in the PR description?

StrikerRUS · 2018-10-02T10:09:26Z

@guolinke

I just notice we don't need the lgamma

Wow, it's great! This fix is for sure much better than symbol versioning. 😃

StrikerRUS · 2018-10-02T10:19:17Z

@guolinke I've added ... || exit -1 to prevent silent usage of inappropriate gcc.

guolinke · 2018-10-02T11:50:56Z

@superbobry

guoke@rdnn-gpu-nv-01:~$ objdump -T lib_lightgbm.so | fgrep GLIBC_
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.4 __printf_chk
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __errno_location
0000000000000000      DO *UND*  0000000000000000  GLIBC_2.2.5 stdout
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 bind
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 ceil
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 socket
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.4 __vfprintf_chk
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 nanosleep
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 accept
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 log
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 recv
0000000000000000  w   DF *UND*  0000000000000000  GLIBC_2.2.5 __cxa_finalize
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strlen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memcmp
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 send
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memset
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_create
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.4 __sprintf_chk
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.4 __snprintf_chk
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 vsnprintf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __assert_fail
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 nextafter
0000000000000000  w   DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutex_unlock
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strtod
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sqrt
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.14  memcpy
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 connect
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.4 __strcpy_chk
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 __cxa_atexit
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 pow
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fclose
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 inet_ntop
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.4   __stack_chk_fail
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 listen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fflush
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fopen
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 exp
0000000000000000      DO *UND*  0000000000000000  GLIBC_2.2.5 stderr
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.4 __vsprintf_chk
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 strtol
0000000000000000  w   DF *UND*  0000000000000000  GLIBC_2.2.5 pthread_mutex_lock
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fread
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 log2f
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 log2
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 memmove
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3.4 __fprintf_chk
0000000000000000  w   DF *UND*  0000000000000000  GLIBC_2.2.5 __pthread_key_create
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3   __tls_get_addr
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3   freeifaddrs
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 setsockopt
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 sqrtf
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.3   getifaddrs
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 inet_pton
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 fwrite
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 close
0000000000000000      DF *UND*  0000000000000000  GLIBC_2.2.5 logf

superbobry · 2018-10-03T09:23:52Z

Should we expect a release following this PR?

guolinke · 2018-10-03T11:37:37Z

yeah, refer to #1727

dowgrade gcc version on Azure

ecd85c8

StrikerRUS requested a review from guolinke September 30, 2018 17:41

guolinke reviewed Oct 1, 2018

View reviewed changes

small fixes, update Artifacts condition for easy debug.

138c50a

guolinke added 5 commits October 1, 2018 11:37

Update CMakeLists.txt

dea500d

Update CMakeLists.txt

fd38d59

Update CMakeLists.txt

06394e1

Update setup.sh

98d76af

Update CMakeLists.txt

0b216c1

guolinke reviewed Oct 1, 2018

View reviewed changes

Update .vsts-ci.yml

606eeef

StrikerRUS commented Oct 1, 2018

View reviewed changes

Update CMakeLists.txt

3f94c37

guolinke mentioned this pull request Oct 1, 2018

GLIBC Version microsoft/SynapseML#335

Closed

guolinke added 2 commits October 2, 2018 09:31

remove the lgamma

211a9e8

update comment

5c602b6

make sure that we use appropriate compiler

a1c72b8

guolinke approved these changes Oct 2, 2018

View reviewed changes

guolinke merged commit 52dbc57 into master Oct 3, 2018

StrikerRUS deleted the glibc branch October 3, 2018 10:08

StrikerRUS mentioned this pull request Oct 8, 2018

update LightGBM Kaggle/docker-python#320

Merged

superbobry mentioned this pull request Oct 16, 2018

New release with LightGBM 2.2.1 microsoft/SynapseML#390

Closed

StrikerRUS mentioned this pull request Apr 11, 2019

lightGBM 2.2.200 still has glibc issue /lib64/libstdc++.so.6: version : GLIBCXX_3.4.20' not found #1945

Closed

lock bot locked as resolved and limited conversation to collaborators Mar 11, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ci] dowgrade gcc version for Linux on Azure #1718

[ci] dowgrade gcc version for Linux on Azure #1718

StrikerRUS commented Sep 30, 2018 •

edited

StrikerRUS commented Sep 30, 2018 •

edited

guolinke commented Oct 1, 2018

guolinke Oct 1, 2018

StrikerRUS Oct 1, 2018 •

edited

guolinke commented Oct 1, 2018

guolinke Oct 1, 2018

superbobry Oct 1, 2018

guolinke Oct 1, 2018

superbobry Oct 1, 2018

guolinke Oct 2, 2018

StrikerRUS Oct 1, 2018

guolinke Oct 1, 2018

StrikerRUS Oct 1, 2018

guolinke commented Oct 1, 2018

StrikerRUS commented Oct 1, 2018

guolinke commented Oct 2, 2018 •

edited

superbobry commented Oct 2, 2018

StrikerRUS commented Oct 2, 2018

StrikerRUS commented Oct 2, 2018

guolinke commented Oct 2, 2018

superbobry commented Oct 3, 2018

guolinke commented Oct 3, 2018 •

edited by StrikerRUS

[ci] dowgrade gcc version for Linux on Azure #1718

[ci] dowgrade gcc version for Linux on Azure #1718

Conversation

StrikerRUS commented Sep 30, 2018 • edited

StrikerRUS commented Sep 30, 2018 • edited

guolinke commented Oct 1, 2018

Choose a reason for hiding this comment

StrikerRUS Oct 1, 2018 • edited

Choose a reason for hiding this comment

guolinke commented Oct 1, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guolinke commented Oct 1, 2018

StrikerRUS commented Oct 1, 2018

guolinke commented Oct 2, 2018 • edited

superbobry commented Oct 2, 2018

StrikerRUS commented Oct 2, 2018

StrikerRUS commented Oct 2, 2018

guolinke commented Oct 2, 2018

superbobry commented Oct 3, 2018

guolinke commented Oct 3, 2018 • edited by StrikerRUS

StrikerRUS commented Sep 30, 2018 •

edited

StrikerRUS commented Sep 30, 2018 •

edited

StrikerRUS Oct 1, 2018 •

edited

guolinke commented Oct 2, 2018 •

edited

guolinke commented Oct 3, 2018 •

edited by StrikerRUS