hunpos binaries do not work for modern linux kernels (3.x)

[reckart]

The HunPosTagger does not work if it is deployed as maven shaded-jar file.
The binary loading fails. It seems that the created temporary folder with the binary
is destroyed before the file is used.

Caused by: java.io.IOException: Cannot run program "/tmp/dkpro8956676736362431887runtime/hunpos-tag":
error=2, Datei oder Verzeichnis nicht gefunden
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1048)
        at de.tudarmstadt.ukp.dkpro.core.hunpos.HunPosTagger.process(HunPosTagger.java:186)
        ... 15 more
Caused by: java.io.IOException: error=2, Datei oder Verzeichnis nicht gefunden
        at java.lang.UNIXProcess.forkAndExec(Native Method)
        at java.lang.UNIXProcess.<init>(UNIXProcess.java:248)
        at java.lang.ProcessImpl.start(ProcessImpl.java:134)
        at java.lang.ProcessBuilder.start(ProcessBuilder.java:1029)
        ... 16 more

I looked into the code, but the usage of the RuntimeProvider seems to be correct (e.g.
the reference should remain valid once initialized). The problem does not occur if
run from within Eclipse, but only in jar mode.

Anyone an idea what the problem could be?

Original issue reported on code.google.com by Tobias.Horsmann on 2015-05-01 17:32:10

[reckart]

I think this may be a variant of issue 107: Fails to run the binary, with "noexec" /tmp

Can you please check that the /tmp file system on the machine where you are running
the shaded jar is not mounted with the "noexec" flag.

It would probably be a good idea to port the fix for 107 to the RuntimeProvider.

Original issue reported on code.google.com by richard.eckart on 2015-05-02 11:29:44

• Labels added: Module-api.resources, DKPro-ASL
• Labels removed: Module-hunpos

[reckart]

@Horsmann did you ever get around to check this?

[Horsmann]

Sorry, no I did not.
How do I see the settings of the temporary folder? I tried the "mount" command, but I does not list any temporary folders?

[reckart]

Hm, check your fstab: http://www.cyberciti.biz/faq/linux-add-nodev-nosuid-noexec-options-to-temporary-storage-partitions/

I guess if your tmp folder is not mounted as a separate file system in fstab, then noexec may not be the reason for the problem. Make sure to check on the right machine: the one where you were trying to run the standalone ;)

[Horsmann]

I am on the right machine ^^

I am not the admin on this machine, but I assume /export/data is in my case the location which also contains the temporary directory (I find nothing else that sound like a temporary directory..)
It is set to "auto" which maps to "default" which should grant "exec"

Btw, I just tested the jar on OS-X and this error does not occur.

[Horsmann]

Was the previous comment helpful to you to understand the problem.

[reckart]

The temporary directory should be "/tmp" - it should not be under "/export/data". Check your environment variables to see where your temp dir resides. I still have the feeling this is a "no_exec" problem.

[Horsmann]

The temp directory has this "ls" output: The creator should be able to do everything
lrwxr-xr-x@

[Horsmann]

I use the same machine that I use also use for other binaries that are loaded with the RuntimeProvider into temp directory where they are executed (CRFSuite for instance). If this is the cause of the problem the other binaries should share the problem?

[Horsmann]

I talked to my sysadmin of the compute server, he confirmed that the temp directory is executable. I can't tell why the CRFsuite binary works, but not this one.

[reckart]

Did you try copying the raw hunpos binary to the machine and simply try to run it on the command line?
There might be some shared-library missing. E.g. on a 64bit machine, the 32bit compatibility libraries might not be present.

[Horsmann]

I just tried both neither 64bit nor 32bit binaries are runnable on my compute server. (i.e. -bash: ./hunpos-tag: Datei oder Verzeichnis nicht gefunden)
The missing shared libraries sound promising.

[reckart]

Use ldd to check the libraries, e.g. ldd ./hunpos-tag.

[Horsmann]

hm,
"Das Programm ist nicht dynamisch gelinkt"

The binaries for linux might be just broken?

[reckart]

hm, no - then it is a static library that should no depend on any shared libraries.

[Horsmann]

Is there anyone with linux as OS that has ever run HunPos - I am all Mac, you are too, right?

[reckart]

well, the unit tests on Jenkins are on Linux.

[reckart]

Did you actually "chmod +x" the executable before trying to run it?

[Horsmann]

yes. dunno why it is not working in my case? Does Jenkins maybe use a different version of the binaries?

[reckart]

It uses the binaries from zoidberg from the Maven artifact. You can try to download the binaries from the hunpos homepage just to be sure and see if they run on your server.

https://code.google.com/p/hunpos/

[reckart]

@giuliopaci Forget it - found it: https://github.com/mivoq/hunpos/releases/tag/v0.0.20170218

[ecamaj]

@giuliopaci Using it from dkpro core (https://dkpro.github.io/dkpro-core/).
Just put it in Docker (tried ubuntu/centos7), with java8, and it said it cannot find hunpos-tag file (generated in /tmp/ by dkpro core)

However - I just made it work by @reckart suggestion! Bravo @reckart! :)

Installed inside docker (ubuntu) 32 bit libraries by instructions from this link
http://stackoverflow.com/questions/23182765/how-to-install-ia32-libs-in-ubuntu-14-04-lts-trusty-tahr

sudo apt-get update
sudo apt-get install lib32z1 lib32ncurses5 lib32bz2-1.0 lib32stdc++6

and it is working.

Unfortunatelly now model loading by MST parser is never finishing -hangs on this line:

Feb 15, 2017 11:22:05 PM org.glassfish.grizzly.http.server.NetworkListener start
INFO: Started listener bound to [127.0.0.1:8080]
Feb 15, 2017 11:22:05 PM org.glassfish.grizzly.http.server.HttpServer start
INFO: [HttpServer] Started.
Feb 15, 2017 11:22:05 PM Server main
INFO: API server started
Press any key to stop the server...
Feb 15, 2017 11:22:10 PM de.tudarmstadt.ukp.dkpro.core.api.resources.internal.ApacheCommonsLoggingAdapter info
INFO: :: loading settings :: url = jar:file:/root/.m2/repository/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Feb 15, 2017 11:22:10 PM de.tudarmstadt.ukp.dkpro.core.api.resources.internal.ApacheCommonsLoggingAdapter info
INFO: :: loading settings :: url = jar:file:/root/.m2/repository/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Feb 15, 2017 11:22:11 PM de.tudarmstadt.ukp.dkpro.core.api.resources.ResourceObjectProviderBase configure
INFO: Producing resource from jar:file:/root/.ivy2/cache/de.tudarmstadt.ukp.dkpro.core/de.tudarmstadt.ukp.dkpro.core.hunpos-upstream-tagger-hr-mte5.defnpout/jars/de.tudarmstadt.ukp.dkpro.core.hunpos-upstream-tagger-hr-mte5.defnpout-20130509.jar!/de/tudarmstadt/ukp/dkpro/core/hunpos/lib/tagger-hr-mte5.defnpout.model
Feb 15, 2017 11:22:11 PM de.tudarmstadt.ukp.dkpro.core.api.resources.ResourceObjectProviderBase loadResource
INFO: Producing resource took 27ms
model loaded
tagger compiled
Feb 15, 2017 11:22:11 PM de.tudarmstadt.ukp.dkpro.core.api.resources.internal.ApacheCommonsLoggingAdapter info
INFO: :: loading settings :: url = jar:file:/root/.m2/repository/org/apache/ivy/ivy/2.3.0/ivy-2.3.0.jar!/org/apache/ivy/core/settings/ivysettings.xml
Feb 15, 2017 11:22:11 PM de.tudarmstadt.ukp.dkpro.core.api.resources.ResourceObjectProviderBase configure
INFO: Producing resource from [jar:file:/root/.ivy2/cache/de.tudarmstadt.ukp.dkpro.core/de.tudarmstadt.ukp.dkpro.core.mstparser-upstream-parser-hr-mte5.defnpout/jars/de.tudarmstadt.ukp.dkpro.core.mstparser-upstream-parser-hr-mte5.defnpout-20130527.jar!/de/tudarmstadt/ukp/dkpro/core/mstparser/lib/parser-hr-mte5.defnpout.model] redirected from [jar:file:/root/.ivy2/cache/de.tudarmstadt.ukp.dkpro.core/de.tudarmstadt.ukp.dkpro.core.mstparser-model-parser-hr-mte5.defnpout/jars/de.tudarmstadt.ukp.dkpro.core.mstparser-model-parser-hr-mte5.defnpout-20130527.1.jar!/de/tudarmstadt/ukp/dkpro/core/mstparser/lib/parser-hr-mte5.defnpout.properties]
Feb 15, 2017 11:22:11 PM de.tudarmstadt.ukp.dkpro.core.mstparser.MstParser createOptions(412)
INFO: Using model order (mstparser.param.order): 2
Feb 15, 2017 11:22:11 PM de.tudarmstadt.ukp.dkpro.core.mstparser.MstParser loadParser(436)
INFO: Retrieving model

and after running another request (made AnalysisEngine object static so that model does not reloads and put everything on web server so that requests could be done on it) it is failing with NullPointerException:

Caused by: java.lang.NullPointerException
	at de.tudarmstadt.ukp.dkpro.core.mstparser.MstParser.process(MstParser.java:267)

I would guess that it is trying to read model which never finished loading in the first request, but can anyone check somehow?

Anyway, hunpos tagging is working like a charm now.

[reckart]

@ecamaj I have no clue about the Croatian language but combining Hunpos with the MSTParser models seem to produce odd output. We have a unit test here. The POS tagset and the tagset encoded in the MstParser model look more-or-less similar. However, the parser produces four ROOT nodes which looks rather wrong. Actually I had to disable the tests because DKPro Core today has sanity checks that make sure that a sentence cannot have more than one dependency ROOT node... If you are familiar with Croatian, maybe you see some obvious problems in the setup? Have you had any success in combining the Croatian hunpos and MstParser models outside of DKPro Core? Btw. it might be a good idea to move this discussion from here to either the mailing list or to a new issue.

[ecamaj]

Yes, that is what my company does - NLP for the Croatian language. We didn't yet spend too much time into analysis of results of MSTParser tagging, but I tried now that unit test example sentence in croatian with manually downloaded model (from its original website) and manually installed MSTparser (from it's original website) and tested it, and yes, the results are the same:

moramo	vrlo	kompliciran	primjer	rečenicu	,	koja	sadrži	što	više	sastojaka	i	ovisnosti	što	je	više	moguće
Vmr1p	Rgp	Agpmsn	N-msn	N-msn	Z	Pi-fsn--n-a	Vmr3s	Pi3n-a--n-nn	Sg	N-mpg	Cc	Vmn	Pi3n-n--n-nn	Vcr3s	Rgc	Agpnsn
Pred	Adv	Atr	Ap	Sb	Punc	Sb	Pred	Atr	Atr	Atr	Co	Pred	Pred	Aux	Adv	Pnom
0	3	4	5	8	8	8	1	11	11	4	10	0	0	8	17	15

Last two lines are result from MST parser, second line is result from hunpos.
Hunpos tagging gives good results, MST parser - bad on the first glance.

Having in mind that your Croatian example is bad Croatian in the first place, the correct sentence would be something like this:

moramo odraditi vrlo kompliciran primjer , rečenicu koja sadrži što više sastojaka i ovisnosti , što je više moguće

Running MSTParser on that corrected sentence gives even more ROOT elements.
We analysed now what might be the problem, and it seems that this "multiple-roots problem" might be linked to the fact that model has been trained on "CONLL-X"(http://anthology.aclweb.org/W/W06/W06-2920.pdf) tagged sentences (as stated on model source site http://nlp.ffzg.hr/resources/models/dependency-parsing/). I also checked the source data that the model has been trained on, and yes, there are multiple 0 dependencies in one sentence.
CONLL-X document says:

7. HEAD: Head of the current token, which is either a value of ID, or zero (’0’) if the token links to the virtual root node of the sentence. Note that depending on the original treebank annotation, there may be multiple tokens with a HEAD value of zero.

So, it seems that the you should cover this use-case in your unit tests and potentially other parts of dkpro core.

Anyway, all this is not connected to the fact that installing IA32 libraries enabled hunpos tagging, but not MSTparser which hangs on "loading model" - could you check that?

Best,
Eduard

[reckart]

@ecamaj I used Google Translate to create the test sentence ;)

Regarding MstParser - the unit tests run. However, I run them with -Xmx5g. I would guess that your experience a hang because the JVM runs out of memory. Try giving it more memory.

[reckart]

I opened a new issue for the ROOTs problem: #1033

[ecamaj]

:) Now you have someone to ask about Croatian.

Yup, that helped! Since I'm running my little test program through mvn exec, this helped:
export MAVEN_OPTS=-Xmx5g

Everything works now.

We should work together, kicking problems without sweat. :)

[reckart]

Thanks :)

I opened a new issue for the ROOTs problem: #1033

[Horsmann]

I pulled hunpos from Git and gave it a try on our 4.4 Linux kernel. From command line training and executing the trained model seems to work. New windows binaries are provided on the website. I can locally build linux 64bit and osx. The only binary someone else would have to build is the linux 32 bit version. Can we update the DKPro binaries then?

[reckart]

Do you know if the new binaries can read the old models?

[Horsmann]

No, and I would have to figure out how to static link with ocalm - you know this by any chance?

[reckart]

I'm afraid I'm not really familiar with ocaml. Maybe @giuliopaci has an idea?

[reckart]

By "no" you mean you don't know if the existing models work or the existing models don't work? It would also be a good idea to test if the models work with the different binaries for the different platforms.

[Horsmann]

I don't know if they work.

[Horsmann]

The German model from the .jar does still work so I assume the others work as well.

[Horsmann]

I just noticed that the current DKPro binaries are also non-static:

ldd gives me this on hunpos-tag:
linux-gate.so.1 => (0xf77e2000)
libm.so.6 => /lib32/libm.so.6 (0xf7789000)
libdl.so.2 => /lib32/libdl.so.2 (0xf7784000)
libc.so.6 => /lib32/libc.so.6 (0xf75d9000)
/lib/ld-linux.so.2 (0x56599000)

my recompiled version has almost the same dependencies:
linux-vdso.so.1 => (0x00007ffc0b3ee000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f937097e000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f937077a000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f93703b4000)
/lib64/ld-linux-x86-64.so.2 (0x00005581e0adf000)

Are hose dependencies ok? By what I read this is stuff that should be there anyway? If I can use non-static binaries this would be a lot easier to prepare. I am not really sure how to hack into the build-script that I want static linked files.

[reckart]

These dependencies should be ok.

[Horsmann]

@reckart I cannot reproduce the original problem anymore. I tried running HunPos from a .jar on 1.7.0 and 1.8.0 which both works. I am on a linux 4.4.0-66-generic kernel now and do not have a 3.x kernel accessible anymore. Might it be that the problem is limited to 3.x?
I have no means to test if the new binaries do actually fix the problem... do you still have a 3.x kernel setup to test the new binaries?

[reckart]

Our Jenkins build server is a Linux 3.2.0 and there it seems to be working as well.

[reckart]

So what should we do here?

• close the issue as not a problem?
• close the issue as fixed?
• resolve it for 1.9.0 - if yes, what needs to be done?
• defer it to 1.10.0?

[Horsmann]

I would say close it for the moment as not a problem (in many situations). The problem is still there but the circumstances in which in occurs are unknown. The problem occurs seemingly sometimes for some (older?) kernel versions.

[reckart]

Ok, well, then let's close it for the time being. If somebody stumbles over this again and cannot solve it be installing the 32-bit compatibility libraries, then we can reopen it.

[reckart]

We now have a second Debian build node where the hunpos does not run :/

• Linux frankie 3.16.0-4-amd64 #1 SMP Debian 3.16.51-3 (2017-12-13) x86_64 GNU/Linux
• libc6 2.24-11+deb9u3

[reckart]

We upgraded the build node to a new kernel, but hunpos still does not run:

• Linux frankie 4.16.0-0.bpo.2-amd64 #1 SMP Debian 4.16.16-2~bpo9+1 (2018-06-26) x86_64 GNU/Linux
• libc6 2.24-11+deb9u3