Karaf client take a very long time/hangs if you to execute multiple commands on 3.2M3

[boc-tothefuture]

Not sure if this goes back before M3 - but it is not present in 3.0 or 3.1.

Steps to reproduce:

1. Download OpenHAB 3.2M3
2. Start openhab, i.e. ./start.sh
3. Loop over or execute multiple commands using karaf

Bash script demonstrating problem:

commands=(foo bar baz qux qaz quu)

for i in "${commands[@]}"
do
  echo $i
  time ./runtime/bin/client -p habopen "log:set trace $i"
done

Sometimes they are quick, other times, very slow (minutes) and they just timeout. Ctrl-C doesn't seem to interrupt.

[jimtng]

I'm seeing the same - sometimes it takes 2 minutes+

[hmerk]

As we are now on 3.2.0.M4, could you please revalidate on that version.

[jimtng]

As we are now on 3.2.0.M4, could you please revalidate on that version.

I've just tried this on 3.2.0-SNAPSHOT - Build openhab/openhab-core#2554, and the problem remains.

[boc-tothefuture]

@hmerk Is this considered blocking for the 3.2 release?

Anyone who has automation written against the openhab karaf client is going to have failures.

[hmerk]

@boc-tothefuture Im am not involved in release planning, so cannot comment.
@wborn Any idea what could cause those delays?

[wborn]

Karaf was upgraded so most likely it is some regression in Karaf or its dependencies. So to get it fixed you probably want to test if it is reproducible on a bare Karaf and if so create an issue in the Karaf issue tracker.

[hmerk]

@wborn As this was not an issue before Karaf upgrade, shouldn‘t we revert #1315 ?

[wborn]

Maybe it is fixed in the forthcoming 4.3.4 release. If you hurry up with reporting the issue it might even get fixed in it. 😉

[wborn]

Sometimes they are quick, other times, very slow (minutes) and they just timeout. Ctrl-C doesn't seem to interrup

I cannot reproduce this as it is always quick for me (tested 100+ times on Ubuntu 20.04.3).
Perhaps it is only an issue for a certain OS/Java version?

[jimtng]

Mine is running inside a docker on Centos 8, and I can reproduce this quite consistently. Last tried on 3.2.0-SNAPSHOT - Build #2582

# Executed inside the docker
/openhab/conf/scripts# bash slow.sh
foo

real	0m2.278s
user	0m1.669s
sys	0m0.131s
bar

real	2m1.259s
user	0m1.628s
sys	0m0.136s
baz

real	0m1.521s
user	0m1.514s
sys	0m0.104s
qux

real	2m1.221s
user	0m1.563s
sys	0m0.140s
qaz

real	0m1.678s
user	0m1.550s
sys	0m0.107s
quu

real	0m1.411s
user	0m1.475s
sys	0m0.109s

[boc-tothefuture]

I can reproduce it on the latest snapshot on Mac OS.

[boc-tothefuture]

I opened a defect on the Karaf tracker, here: https://issues.apache.org/jira/browse/KARAF-7288

I have verified this behavior does not happen on 4.3.2. @wborn can we roll back the upgrade to 4.3.3 until Karaf team resolves the defect?

[boc-tothefuture]

@wborn looks like they are aware of the issue and they have fixed it in 4.3.4. However, the timeline for 4.3.4 is beyond the 'normal?' date for which openhab gets released. So, can we roll back the upgrade to 4.3.3 and then test with 4.3.4 when it is released?

[wborn]

So, can we roll back the upgrade to 4.3.3 and then test with 4.3.4 when it is released?

While it may fix this one issue, it will surely reintroduce many other regressions that are considered to be fixed now.

[boc-tothefuture]

@wborn are we aware of any workarounds? This has a significant impact on the JRuby libraries as it will prevents testing.

[jimtng]

@boc-tothefuture would it be possible to replace karaf jar with 4.3.2 within our testing environment?

[wborn]

I don't know of any workarounds. It wasn't even reproducible for me.

Most likely Karaf 4.3.4 will be released within a few weeks. The Karaf release schedule is very unpredictable and it is also common that projected release dates are not met.

[boc-tothefuture]

Karaf team says there is not a workaround: https://issues.apache.org/jira/browse/KARAF-7288

@jimtng I don't know if we can just replace the karaf jar, I imagine it sits on top of a bunch of other dependencies.

@wborn I know you didn't hit the bug - but anyone who has any scripting against the karaf client is going to be impacted by this if OH 3.2 releases with this version of karaf with no known workarounds. I don't know the size of that population.

[boc-tothefuture]

This SO has some pointers on how to execute rest commands: https://stackoverflow.com/questions/54198755/how-to-get-apache-karaf-bundles-information-via-web-services-rest-soap

But we would have to find a way to install war support (the example in SO doesn't work for me) and the war file without using the karaf client.