Is this statically linking with tpie?

[pombredanne]

I was wondering if tpie is linked statically or dynamically witth the rest of keyvi?
Based on

keyvi/CMakeLists.txt

Line 60 in c7b523b

set(_KEYVI_LINK_LIBRARIES_STATIC "${_KEYVI_LINK_LIBRARIES_STATIC} tpie")

it looks like it is a static link. Am I correct?
Thanks!

[pombredanne]

FYI see also this thomasmoelhave/tpie#242

[hendrikmuhs]

Yes, it's linked statically. Otherwise it's hard to ship the distribution. We have a disclaimer in the README about "Licence and 3rdparty dependencies", however it's not very good at explaining the issue.

Afaik there is no commercial product build on top of keyvi, at least not that I am aware of. Still, we should probably make the licensing issue more clear. The keyvi source code is apache, but built binaries or not free of use according to apache 2, because of included 3rd party licenses. I think only TPIE raises the restrictions, the other 3rdparty depedencies do not put additional restrictions. Best to my knowledge the problem only applies to software that ships, but not SaaS. However, I am not a lawyer, this is only what I think.

Having that said, TPIE is not an essential part of keyvi. We only use the (external) sorter. It's possible to create TPIE-free binaries. This would restrict the size of keyvi dictionaries you can build from scratch. The other sorter works in-memory, so all data must fit in memory. However there is also the possibility to build large dictionaries in stages by building n smaller ones and than use keyvimerger to merge them to 1.

I guess you have a concrete reason to ask, I happily discuss the options with you (If you don't want to discuss in public, you find my email address easily).

[pombredanne]

Thank you for the detailed reply!

You wrote:

Yes, it's linked statically. Otherwise it's hard to ship the distribution.

That's OK by me to have multiple shared objects.

FWIW I am considering keyvi for use in Apache-licensed projects at https://github.com/nexB such as https://github.com/nexB/scancode-toolkit (that happens to deal with license detection) that I would like to keep Apache-licensed. So this is all for FOSS code alright. And as an occupational hazard I always scan for license any new code I look at with https://github.com/nexB/scancode-toolkit :) hence the question.
FWIW, ATM I use https://github.com/WojciechMula/pyahocorasick (which is in C++) (where I contribute) for certain search tasks, and I am intrigued by your FST approach.

[hendrikmuhs]

For a matching use case my gut feeling: you probably don't need external memory sort. Do you have a rough estimate regarding how much data we talk about?

Talking about possible solutions, its hard to change the default package short term. Taking away the ability to build super large keyvi files would be a step backwards. But if you build the package on your own, it's possible to add the ability to build a TPIE-free package or a TPIE dynamically linked package.

Long term we want to get rid of TPIE. This is not, because we are unsatisfied, but we only use a tiny part and pull in a large dependency for it. The 2nd reason is the licensing issue.

[hendrikmuhs]

I just realized, you can already build without TPIE using make KEYVI_DISABLE_TPIE=1. This still builds TPIE and it links with it, but the linker is smart enough to drop binary code it does not require. The created binaries are smaller in size and a quick analysis with objdump shows no TPIE code. However, I do not guarantee anything, it would be cleaner to integrate the above into the cmake build file and ensure it is neither build nor linked.

[pombredanne]

@hendrikmuhs Thank you++ for checking this!

For a matching use case my gut feeling: you probably don't need external memory sort. Do you have a rough estimate regarding how much data we talk about?

These may be quite large alright and would surely not always fit in memory.

Talking about possible solutions, its hard to change the default package short term. Taking away the ability to build super large keyvi files would be a step backwards. But if you build the package on your own, it's possible to add the ability to build a TPIE-free package or a TPIE dynamically linked package.

Long term we want to get rid of TPIE. This is not, because we are unsatisfied, but we only use a tiny part and pull in a large dependency for it. The 2nd reason is the licensing issue.

I am fine with tpie if I can build tpie as a shared library optionally.

[hendrikmuhs]

@pombredanne

FYI I removed the tpie dependency, we will make a new release asap. This should solve your issue.

[pombredanne]

Thank you ++ 🙇