Layer script not including all libraries #14
Comments
|
@meyer1994 thanks again for the issue. The only solution I can think of is to move or build those libraries at build time to I don't have time this week to work on this sadly |
|
@meyer1994 in fact it goes down to understand what libraries are available in lambda runtime. we are using Thus we just need to copy in I'll continue to investigate |
|
@meyer1994 just run the only differences I see when running this in What we can see is first, we might be missing |
|
@meyer1994 I cannot reproduce the error you mentioned, can you share more about how you created your layer or pacakge ? |
|
I am trying to reproduce it, unsuccessfully. I will try to create a reproducible example in the next couple of days. Weirdly, my stack is working based on waht I said. It wasn't working without copying the Weirdly, again, I have not copied the |
|
after other checks it seems that libxml2 is present on lambda runtime env 🤔. |
|
@meyer1994 with the latest update I don't see how this could still be a problem. Can you test with the latest lambda layer or docker images and close here if everything works? |
|
Weird. The error persists :( Just tried with the latest layer: When trying to open Sample lambda: from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
def handler(event, context):
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount()) |
|
@meyer1994 thanks |
|
You sure it works fine? I am running it on |
|
I've try with |
|
I tried somethings around, with no avail... It seems to me that adding the libraries to the layer by hand, using: Is the best solution for now. If you want you can close this issue and reopen it when it seems fit. |
|
🤔 but it seems to me that are you using python3.7 ? |
|
I am using python3.7.
Yes, it is. But it is in the wrong place. When you add the file to the layer, it will be unzipped in Using this command in the layer creation: $ zip -r9 --symlinks /tmp/${PACKAGE_NAME}.zip /lib64/libexpat.so*After the layer is deployed into the lambda, the library will be in |
we don't ship
|
Yes yes. I was trying to say when someone adds the file to the layer (like I did). My bad.
I don't know what else to do. For now, my version is working with the copied |
|
Ok. I am ashamed. The new layer works!!! The problem was with IAM roles... Adding permissions to access sentinel bucket made it work fine. See service: layertest
provider:
name: aws
runtime: python3.7
stage: dev
versionFunctions: false
region: eu-central-1
environment:
PROJ_LIB: '/opt/share/proj'
GEOS_CONFIG: '/opt/bin/geos-config'
GDAL_DATA: '/opt/share/gdal'
GDAL_CONFIG: '/opt/bin/gdal-config'
GDAL_CACHEMAX: '512'
GDAL_HTTP_MERGE_CONSECUTIVE_RANGES: 'YES'
GDAL_HTTP_MULTIPLEX: 'YES'
GDAL_HTTP_VERSION: '2'
GDAL_DISABLE_READDIR_ON_OPEN: 'EMPTY_DIR'
VSI_CACHE: 'true'
VSI_CACHE_SIZE: '536870912'
CPL_TMPDIR: '/tmp'
CPL_VSIL_CURL_ALLOWED_EXTENSIONS: '.tif,.jp2,.gml'
iamRoleStatements:
- Effect: Allow
Action:
- s3:GetObject
Resource:
- arn:aws:s3:::sentinel-s2-l1c*
package:
include:
- handler.py
functions:
layer:
handler: handler.handler
layers:
- arn:aws:lambda:eu-central-1:524387336408:layer:gdal30-py37-geo:5And import glob
from osgeo import gdal, ogr
gdal.UseExceptions()
ogr.UseExceptions()
gdal.SetConfigOption('AWS_DEFAULT_REGION', 'eu-central-1')
gdal.SetConfigOption('AWS_REQUEST_PAYER', 'requester')
BUCKET = 'sentinel-s2-l1c'
KEY = 'tiles/22/J/GQ/2019/8/3/0/qi/MSK_CLOUDS_B00.gml'
def handler(event, context):
libs = glob.glob('/opt/libs*/*')
libs = sorted(libs)
print('\n'.join(libs))
key = f'/vsis3/{BUCKET}/{KEY}'
mask = ogr.Open(key)
print('Layers:', mask.GetLayerCount())
layer = mask.GetLayer()
print('Features:', layer.GetFeatureCount())After deploying: |
|
🎉 nothing to be ashamed, this happened to me so many time 😄 |
After PR #12 got merged, I tried to create a layer with the script located here. But I still got the same errors:
After some intense search I've stumbled upon this doc which led me to
ldd.From inside the docker image I ran
$ ldd /opt/lib/libgdal.so. Lo and behold, lots of info came out of it:It appear that the build process for gdal does not place every library inside the
/opt/libdir. So, when creating the layer, we should add the libraries by hand (unfortunately).The addition of the following line to the
create-lambda-layer.shfile fixed the error for me:However, I think there are more problems that may appear from this problem. For example,
libsqlite3.so.0not included by the script as well. I have not tested it. But it is entirely possible.I did not make a PR because I am not sure how to fix this in a more generic way. Maybe adding each lib by hand is the best option, but I am not sure.
The text was updated successfully, but these errors were encountered: