Issue with curl version #7

bu-ra3y · 2019-01-22T22:12:16Z

When using the 2.3.2 and 2.4.0 images I get this error when using Rasterio to open a TIF on S3:
CPLE_AppDefined in GDAL was built against curl 7.59.0, but is running against 7.51.0. Runtime failure is likely !

I do not hit that error when using your old 2.2.2 image.

Does this have to do with the curl version you specify here?

amazonlinux-gdal/Dockerfile

Line 31 in b2ca47b

CURL_VERSION=7.59.0 \

The text was updated successfully, but these errors were encountered:

vincentsarago · 2019-01-23T03:04:49Z

@bu-ra3y 👋 thanks for opening the issue.
I also noticed the log message on some AWS Lambda process.

Can you paste the full code to reproduce the message ?

Thanks

vincentsarago · 2019-01-23T03:45:37Z

Alright I figured this out @bu-ra3y

So on AWS Lambda the LD_LIBRARY_PATH is set to be /lib64:/usr/lib64:$LAMBDA_RUNTIME_DIR:$LAMBDA_RUNTIME_DIR/lib:$LAMBDA_TASK_ROOT:$LAMBDA_TASK_ROOT/lib:/opt/lib and on default aws lambda image we can found libcurl in /usr/lib64

	libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007ff117fdc000)

The problem is then we ship our own curl lib with GDAL located in /usr/task/lib with is in the LD_LIBRARY_PATH default but way after /usr/lib64 so by default AWS will always ignore our own curl lib.

To fix this I think you need to set LD_LIBRARY_PATH=/var/task/lib/ in the lambda environment options.

cc @mojodna and @fredliporace

vincentsarago · 2019-01-23T04:05:13Z

To fix this I think you need to set LD_LIBRARY_PATH=/var/task/lib/ in the lambda environment options.

This will need to be tested but it seems to me that when setting the LD_LIBRARY_PATH gdal is linked to less libraries (ldd output). I'm not linux expert so I'd love to have other 👀 on this matter.

Default env

$ ldd /var/task/lib/libgdal.so
tlinux-vdso.so.1 =>  (0x00007fff17bfe000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f6f42fb4000)
libproj.so.13 => /var/task/lib/libproj.so.13 (0x00007f6f42d3d000)
libgeos_c.so.1 => /var/task/lib/libgeos_c.so.1 (0x00007f6f42b0c000)
libwebp.so.7 => /var/task/lib/libwebp.so.7 (0x00007f6f4289f000)
libopenjp2.so.7 => /var/task/lib/libopenjp2.so.7 (0x00007f6f42645000)
libjpeg.so.62 => /usr/lib64/libjpeg.so.62 (0x00007f6f423f2000)
libzstd.so.1 => /var/task/lib/libzstd.so.1 (0x00007f6f42173000)
libz.so.1 => /lib64/libz.so.1 (0x00007f6f41f5d000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f6f41d41000)
librt.so.1 => /lib64/librt.so.1 (0x00007f6f41b39000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f6f41935000)
libcurl.so.4 => /usr/lib64/libcurl.so.4 (0x00007f6f416be000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f6f413b9000)
libm.so.6 => /lib64/libm.so.6 (0x00007f6f410b7000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f6f40ea1000)
libc.so.6 => /lib64/libc.so.6 (0x00007f6f40add000)
/lib64/ld-linux-x86-64.so.2 (0x00007f6f4424c000)
libgeos-3.6.2.so => /var/task/lib/libgeos-3.6.2.so (0x00007f6f4072f000)
libidn2.so.0 => /usr/lib64/libidn2.so.0 (0x00007f6f4050e000)
libssh2.so.1 => /usr/lib64/libssh2.so.1 (0x00007f6f402e6000)
libpsl.so.0 => /usr/lib64/libpsl.so.0 (0x00007f6f40071000)
libssl3.so => /usr/lib64/libssl3.so (0x00007f6f3fe1f000)
libsmime3.so => /usr/lib64/libsmime3.so (0x00007f6f3fbf8000)
libnss3.so => /usr/lib64/libnss3.so (0x00007f6f3f8d3000)
libnssutil3.so => /usr/lib64/libnssutil3.so (0x00007f6f3f6a4000)
libplds4.so => /lib64/libplds4.so (0x00007f6f3f4a0000)
libplc4.so => /lib64/libplc4.so (0x00007f6f3f29b000)
libnspr4.so => /lib64/libnspr4.so (0x00007f6f3f05d000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f6f3ee0f000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f6f3eb28000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f6f3e8f6000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f6f3e6f3000)
liblber-2.4.so.2 => /lib64/liblber-2.4.so.2 (0x00007f6f3e4e4000)
libldap-2.4.so.2 => /lib64/libldap-2.4.so.2 (0x00007f6f3e292000)
libunistring.so.0 => /usr/lib64/libunistring.so.0 (0x00007f6f3df7c000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f6f3dd0e000)
libicuuc.so.50 => /usr/lib64/libicuuc.so.50 (0x00007f6f3d998000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f6f3d789000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f6f3d586000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f6f3d36c000)
libsasl2.so.2 => /usr/lib64/libsasl2.so.2 (0x00007f6f3d151000)
libicudata.so.50 => /usr/lib64/libicudata.so.50 (0x00007f6f3bb7e000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f6f3b95d000)
libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f6f3b726000)
libfreebl3.so => /lib64/libfreebl3.so (0x00007f6f3b524000)

with LD_LIBRARY_PATH: /var/task/lib

$ ldd /var/task/lib/libgdal.so
linux-vdso.so.1 =>  (0x00007ffe08bc7000)
libcrypto.so.10 => /lib64/libcrypto.so.10 (0x00007f1c1ce2f000)
libproj.so.13 => /var/task/lib/libproj.so.13 (0x00007f1c1cbb8000)
libgeos_c.so.1 => /var/task/lib/libgeos_c.so.1 (0x00007f1c1c987000)
libwebp.so.7 => /var/task/lib/libwebp.so.7 (0x00007f1c1c71a000)
libopenjp2.so.7 => /var/task/lib/libopenjp2.so.7 (0x00007f1c1c4c0000)
libjpeg.so.62 => /var/task/lib/libjpeg.so.62 (0x00007f1c1c22f000)
libzstd.so.1 => /var/task/lib/libzstd.so.1 (0x00007f1c1bfb0000)
libz.so.1 => /lib64/libz.so.1 (0x00007f1c1bd9a000)
libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f1c1bb7e000)
librt.so.1 => /lib64/librt.so.1 (0x00007f1c1b976000)
libdl.so.2 => /lib64/libdl.so.2 (0x00007f1c1b772000)
libcurl.so.4 => /var/task/lib/libcurl.so.4 (0x00007f1c1b501000)
libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00007f1c1b1fc000)
libm.so.6 => /lib64/libm.so.6 (0x00007f1c1aefa000)
libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f1c1ace4000)
libc.so.6 => /lib64/libc.so.6 (0x00007f1c1a920000)
/lib64/ld-linux-x86-64.so.2 (0x00007f1c1e0c7000)
libgeos-3.6.2.so => /var/task/lib/libgeos-3.6.2.so (0x00007f1c1a572000)
libnghttp2.so.14 => /var/task/lib/libnghttp2.so.14 (0x00007f1c1a34c000)
libssl.so.10 => /usr/lib64/libssl.so.10 (0x00007f1c1a0de000)
libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2 (0x00007f1c19e90000)
libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x00007f1c19ba9000)
libcom_err.so.2 => /usr/lib64/libcom_err.so.2 (0x00007f1c199a6000)
libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3 (0x00007f1c19774000)
libkrb5support.so.0 => /usr/lib64/libkrb5support.so.0 (0x00007f1c19565000)
libkeyutils.so.1 => /lib64/libkeyutils.so.1 (0x00007f1c19362000)
libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f1c19148000)
libselinux.so.1 => /usr/lib64/libselinux.so.1 (0x00007f1c18f27000)

vincentsarago · 2019-01-23T04:28:06Z

😢 just tried and it's not working

libgdal.so: cannot open shared object file: No such file or directory

vincentsarago · 2019-01-24T04:54:14Z

🎉 after spending the day compiling GDAL I finally found the solution 😄

amazonlinux-gdal/Dockerfile

Lines 104 to 105 in 358ce32

    
           && touch config.rpath \ 
        
           && LDFLAGS="-Wl,-R,$PREFIX/lib" CFLAGS="-O2 -Wl,-S" CXXFLAGS="-O2 -Wl,-S" ./configure \

The idea is to fix shared libraries path using rpath. Sadly there is one major downside of doing so is that gdal won't be able to run if libgdal.so is somewhere else /var/task/lib. I tried to use the $origin variable but didn't succeed yet.

bu-ra3y · 2019-01-24T06:24:09Z

Way to go, Vincent. When do you need libgdal.so in another location?

On Wed, Jan 23, 2019, 10:54 PM Vincent Sarago ***@***.***> wrote: 🎉 after spending the day compiling GDAL I finally found the solution 😄 https://github.com/RemotePixel/amazonlinux-gdal/blob/358ce32609cb64d31c3fd87b8e254e3e67b93db5/Dockerfile#L104-L105 The idea is to *fix* shared libraries path using *rpath*. Sadly there is one major downside of doing so is that gdal won't be able to run if libgdal.so is somewhere else /var/task/lib. I tried to use the $origin variable but didn't succeed yet. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#7 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AIVAziPBXS69p-758kW1KBHYcGiLj5JIks5vGTx3gaJpZM4aNoGC> .

-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Will Barley Schedule a call <https://calendly.com/will-barley> +1.650.575.5445

vincentsarago · 2019-01-24T13:18:17Z

Hi @bu-ra3y

When do you need libgdal.so in another location?

I never so I'll publish this new version like that and then I'll try to play around.

Note: This also fix a previous problem we had with libjpeg being in lib64/. We first assumed we could put the .so in /lib/ but it appeared to me that AmazonLinux image is also providing a libjpeg.so in /usr/lib64 so by default with the old install, gdal was also fetching the libjpeg from AWS and not ours.
cc @mojodna

bu-ra3y · 2019-01-24T16:58:31Z

@vincentsarago , I really appreciate you addressing this.

How do I take advantage of your fix?
Looking at Docker Hub, it looks like your fix may be contained in tag 2.4.0rc.
So I am guessing that either

I need to switch to use Docker Image 2.4.0rc
Or, you I am guessing you might roll your fix into Image 2.4.0 in which case I will pull it again

Is there anything I need to do to change my paths/environment variables or do I leave that the same?

vincentsarago · 2019-01-24T17:00:33Z

@bu-ra3y no worry
I pushed a 2.4.0rc yesterday but I don't think it solved the problem.

I'll push a new 2.4.0 in a few minutes with the fixes

vincentsarago · 2019-01-24T17:36:38Z

sorry @bu-ra3y I'm still checking everything is fine, I've deploy some lambda package using the new image and it seems the code run slower than when using libcurl provided by AWS Lambda

bu-ra3y · 2019-01-24T17:48:15Z

In case this helps you:
I noticed a slowdown as well, prior to your libcurl fix.
I have been using your 2.2.2 image for a long time.
Going to 2.3.2 was a significant slowdown, and in some cases more than double.
2.4.0 sped things back up but still slower than 2.2.2

The main thing I am doing is reading TIFs from S3 using rasterio.open('s3://....').

vincentsarago · 2019-01-24T17:50:04Z

🤔 I may create a gdal2.4.0-light image without curl but I'll test it before

vincentsarago · 2019-01-24T18:40:45Z

here are some result

using gdal2.4.0-light (without curl and jpegturbo) HTTP/1

$ cat list_1000.txt| python lambda_bench_landsat.py LC08_L1TP_018034_20181017_20181031_01_T1 - --stage light
Nb Good: 1000 / 1000
Mean: 333.85487900000004
Median: 313.97
p5: 144.89600000000002 | p95: 562.0905
min: 40.465 | max: 1643.37

using gdal2.4.0 (with curl and jpegturbo) and HTTP/2 enable

$ cat list_1000.txt| python lambda_bench_landsat.py LC08_L1TP_018034_20181017_20181031_01_T1 - --stage http2

Nb Good: 1000 / 1000
Mean: 394.33198
Median: 356.125
p5: 251.35299999999998 | p95: 647.904
min: 84.51 | max: 2284.39

using gdal2.4.0 (with curl and jpegturbo) and HTTP/1.1 enable

$ cat list_1000.txt| python lambda_bench_landsat.py LC08_L1TP_018034_20181017_20181031_01_T1 - --stage http1

Nb Good: 1000 / 1000
Mean: 464.00965
Median: 396.115
p5: 255.80200000000002 | p95: 751.1659999999999
min: 82.68 | max: 10010.18

I kinda feel both options could be good because shipping gdal with its own CURL should reduce the number of HTTP call (GET) which could be a $ saver when working with requester-pays bucket.

I'll go with the plan of pushing both image gdal2.4.0 and gdal2.4.0-light

I could also backup the old gdal2.2.0 if you need it @bu-ra3y

vincentsarago · 2019-01-24T21:28:26Z

I think we are done here. Thanks @bu-ra3y for you help 👼

This was referenced Jan 23, 2019

[WIP] refactor to use lambci docker images #9

Merged

Add nghttp2 to enable http/2 in CURL rasterio/rasterio-wheels#13

Closed

vincentsarago closed this as completed Jan 24, 2019

vincentsarago mentioned this issue Jan 28, 2019

curl 7.54.0 not building as before rasterio/rasterio-wheels#18

Closed

vincentsarago mentioned this issue Mar 12, 2019

release 1.1.0rc2 developmentseed/geolambda#51

Merged

vincentsarago mentioned this issue Jul 12, 2019

Correct paths resulting package.zip #17

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Issue with curl version #7

Issue with curl version #7

bu-ra3y commented Jan 22, 2019

vincentsarago commented Jan 23, 2019

vincentsarago commented Jan 23, 2019

vincentsarago commented Jan 23, 2019 •

edited

Loading

vincentsarago commented Jan 23, 2019

vincentsarago commented Jan 24, 2019

bu-ra3y commented Jan 24, 2019 via email

vincentsarago commented Jan 24, 2019

bu-ra3y commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

bu-ra3y commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

Issue with curl version #7

Issue with curl version #7

Comments

bu-ra3y commented Jan 22, 2019

vincentsarago commented Jan 23, 2019

vincentsarago commented Jan 23, 2019

vincentsarago commented Jan 23, 2019 • edited Loading

vincentsarago commented Jan 23, 2019

vincentsarago commented Jan 24, 2019

bu-ra3y commented Jan 24, 2019 via email

vincentsarago commented Jan 24, 2019

bu-ra3y commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

bu-ra3y commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

vincentsarago commented Jan 24, 2019

using gdal2.4.0-light (without curl and jpegturbo) HTTP/1

using gdal2.4.0 (with curl and jpegturbo) and HTTP/2 enable

using gdal2.4.0 (with curl and jpegturbo) and HTTP/1.1 enable

vincentsarago commented Jan 24, 2019

vincentsarago commented Jan 23, 2019 •

edited

Loading