Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redshift can't open shared object file #19

Open
bhavika opened this issue Jan 11, 2019 · 1 comment
Open

Redshift can't open shared object file #19

bhavika opened this issue Jan 11, 2019 · 1 comment

Comments

@bhavika
Copy link

bhavika commented Jan 11, 2019

I'm trying to install the h3 library in Redshift using Pip and the utility install script installPipModuleAsRedshiftLibrary.sh.

I am building h3 on Ubuntu 16.04. I am able to successfully upload the h3.zip file to my S3 bucket.

However, when I try to run the UDF, I get this error

OSError: /rdsdbdata/user_lib/0/0/288461.zip/h3/out/libh3.so.1: cannot open shared object file: Not a directory

Full stack trace from svl_udf_log:

File "h3.py", line 39, in <module>
File "__init__.py", line 443, in LoadLibrary
    return self._dlltype(name)
File "__init__.py", line 365, in __init__
    self._handle = _dlopen(self._name, mode)```

The UDF I'm trying to use:

```CREATE OR REPLACE function get_hexbin(coord_lat double precision, coord_lon double precision, res INTEGER)
  RETURNS VARCHAR IMMUTABLE
AS $$
  from h3 import h3
  return 'hello world'
$$ LANGUAGE plpythonu; 

While installing h3 locally, I see this message about skipping wheel but then it uploads a zip file to S3 anyway:

Collecting h3
Saved /home/bhavika/Desktop/.h3/h3-3.1.0-cp36-cp36m-linux_x86_64.whl
Skipping h3, due to already being wheel.
@lgr
Copy link

lgr commented May 22, 2019

I have exactly the same issue. Locally the library works well (tested in docker i386/ubuntu:latest). It seems that Redshift doesn't allow loading C libraries from the Python code - or to be more precise, it doesn't allow open any non-Python files within the library. More details are described in this article:

Only one thing is missing! They don’t let you to read files either. If your Python package want to open any file then the package will fail instantly.

Failing library part is the following:

libh3 = cdll.LoadLibrary(libh3_path)

and even after changing the library path to be an absolute path:

libh3 = cdll.LoadLibrary(os.path.abspath(libh3_path))

The loading operation fails anyway. Any clues how to overcome this limitation @zach-data @IanMeyers?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants