How to import module from .so file inside egg file without using an absolute file path?

[Guangyi-Z]

I posted the details on SO.
Thanks all.

http://stackoverflow.com/questions/43274153/python-how-to-import-module-from-so-file-from-egg-file-without-using-an-absol

[pombredanne]

@janfan why do you use an Egg? eggs are over and were never tested here. You should use a wheel IMHO

[pombredanne]

That said, could you give a try to add zip_safe=False, in here

pyahocorasick/setup.py

Line 73 in 547b535

?
and respin your egg build?

[Guangyi-Z]

Ok, I will try to figure out how wheel can fit into my job. I will return here soon. Thank for your advice.

[pombredanne]

@janfan FWIW, it would be best to have a self standing ticket rather than something mostly on SO and a ticket here ;) for sanity

[Guangyi-Z]

@pombredanne Hi there, I am sorry to say that zip_safe=False did not work.

[Guangyi-Z]

@pombredanne I'm back. Now I review the knowledge about wheel, and it's just a distribution instead of a portable runtime, that is, you can't import from a wheel, which is what I need when I use this package on Spark cluster.

[pombredanne]

eggs are on the way out and supporting them is likely problematic, especially eggs with native code like here.
Perhaps you could instead explain what you are trying to achieve in more details? It could be that there are other solutions.

[Guangyi-Z]

@pombredanne If it is like what you said, I am going to turn to other ways. Thanks for your help.

[pombredanne]

@janfan I would be happy to help, but Sparks doc seem to mention that eggs with native are not something that works with their way of importing. And they are working on supporting wheels.

[pombredanne]

@janfan note that you could also just unzip a wheel (or an egg) and put the .so in your path or PYTHONPATH. Then doing import ahocorasick works just fine.

[pombredanne]

because all of it is just a single .so ;)

[pombredanne]

@janfan did using the .so worked for you?

[Guangyi-Z]

@pombredanne No. I modified the env PYTHON_EGG_CACHE so that the .so can be extracted and loaded. However, there is another subsequent problem- the system can not find the libpython.so.10.

I use it for PySpark tasks, and it is difficult to figure the underlying problems. It has something to do with the Spark cluster.

At last I pushed my operator to pip install pyahocorasick on every node in Spark cluster and solved it.

[pombredanne]

Excellent! so you can close this ticket then?