Travis + apt-get install 4store / 4s-httpd returns with httpd.c:1645 couldn't connect "..." #110

Closed
mwjames opened this Issue Jun 24, 2014 · 22 comments

5 participants

@mwjames

Using Travis-CI (which is integrated with GitHub) will successfully install the application apt-get install 4store Setting up 4store (1.1.4-1) [0] but will fail to load the KB after executing:

4s-backend-setup db
4store[3130]: backend-setup.c:185 erased files for KB db
4store[3130]: backend-setup.c:310 created RDF metadata for KB db
4s-backend db
4s-httpd -D -p 8000 db
4store[3140]: httpd.c:1634 4store HTTP daemon v1.1.4 started on port 8000
4store[3141]: httpd.c:1645 couldn't connect to “db”
4store[3140]: httpd.c:1694 child 3141 exited with return code 3
4store[3140]:  1: 4s-httpd() [0x406cd2]
4store[3140]:  2: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fa70722176d]

Running ps auwwx | grep 4s- will only yield

travis    3138  0.0  0.0   9336   928 pts/0    S+   00:53   0:00 grep 4s-
travis    3139  0.0  0.0 107696  1504 ?        S    00:53   0:00 4s-backend db

[0, 1] shows details for the installation and its failure.

[0] https://travis-ci.org/mwjames/SemanticMediaWiki/builds/28274105
[1] https://s3.amazonaws.com/archive.travis-ci.org/jobs/28274106/log.txt

@msalvadores
Collaborator

This is often a firewall issue. Please, try shutting down the firewall (iptables) temporally to see if it works.

@mwjames

try shutting down the firewall (iptables) temporally

sudo iptables -F
sudo iptables -X
sudo iptables -t nat -F
sudo iptables -t nat -X
sudo iptables -t mangle -F
sudo iptables -t mangle -X
sudo iptables -P INPUT ACCEPT
sudo iptables -P FORWARD ACCEPT
sudo iptables -P OUTPUT ACCEPT

Added above commands but the result remains unchanged with couldn't connect to “db” [0].

[0] https://s3.amazonaws.com/archive.travis-ci.org/jobs/28276750/log.txt

@mwjames mwjames referenced this issue in SemanticMediaWiki/SemanticMediaWiki Jul 1, 2014
Merged

4Store (1.1.4) compliance #370

@swh
Collaborator

I'm not really familiar with the environment, does it run 4s-backend, then 4s-httpd immediately? Maybe adding a sleep of say 4 seconds after the backend command will help? I'm not sure offhand how long 4s-http will retry for.

@mwjames

Doing an additional sleep 5 [0] does unfortunately not solve the problem.

Setting up 4store (1.1.4-1) ...

+sudo iptables -F
+4s-backend-setup db

4store[3199]: backend-setup.c:185 erased files for KB db
4store[3199]: backend-setup.c:310 created RDF metadata for KB db

+4s-backend db
+sleep 5
+4s-httpd -D -p 8088 db

4store[3215]: httpd.c:1634 4store HTTP daemon v1.1.4 started on port 8088
4store[3216]: httpd.c:1645 couldn't connect to “db”
4store[3215]: httpd.c:1694 child 3216 exited with return code 3
4store[3215]:  1: 4s-httpd() [0x406cd2]
4store[3215]:  2: /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed) [0x7fdeab63176d]
4store[3215]:  3: 4s-httpd() [0x4072d5]
4store[3218]: httpd.c:1645 couldn't connect to “db”

[0] https://s3.amazonaws.com/archive.travis-ci.org/jobs/30780186/log.txt

@swh
Collaborator

OK, thanks - is there anything in /var/log/messages?

Looking at the error, it looks more like something went bang than a networking / firewall issue.

@szarnyasg

Hello, I also ran into a similar issue, even though I am not using 4s-httpd.

The repository is located at https://github.com/szarnyasg/travis-4s-test, while the Travis build results are at https://travis-ci.org/szarnyasg/travis-4s-test.

This setups only aims at doing a single import from a Turtle file (with 4s-import) followed by a query (with 4s-query). I disabled IPv6 as it is a common source of problems for 4store.

I created a VirtualBox VM based on the Travis cookbook (https://github.com/travis-ci/travis-cookbooks) on my local computer and it managed to run the operations fine.

Any ideas on this?

Thanks,
Gabor

@swh
Collaborator

Can you paste the error log (as above) and the relevant contents of /var/log/messages?

@szarnyasg

The latest job is at [1], /var/log/syslog is printed at [2].

+4s-backend-setup -v my_cluster
4store[3008]: backend-setup.c:186 erased files for KB my_cluster
Creating data directoryCreate segment directoriesCreating data directoryCreate segment directoriesWriting metadata.
4store[3008]: backend-setup.c:337 created RDF metadata for KB my_cluster
+4s-backend my_cluster
+sleep 5
+ps auxw
+grep '4s-backen[d]'
travis    3016  0.0  0.0 109840  1540 ?        S    12:20   0:00 4s-backend my_cluster
+ls -la /var/lib/4store/my_cluster
total 8
drwxr-xr-x 4 travis travis 120 Aug 30 12:20 .
drwxr-xr-x 3 travis travis  60 Aug 30 12:20 ..
drwxr-xr-x 3 travis travis 240 Aug 30 12:20 0000
drwxr-xr-x 3 travis travis 240 Aug 30 12:20 0001
-rw------- 1 travis travis 835 Aug 30 12:20 metadata.nt
-rw-r--r-- 1 travis travis   9 Aug 30 12:20 runtime.info
+4s-import my_cluster railway-xform-1.ttl --format turtle --verbose
4store[3020]: 4s-import.c:171 couldn't connect to “my_cluster”
+4s-query my_cluster 'SELECT ?x ?y ?z WHERE { ?x ?y ?z }' -f text --verbose
+head -n 20
4store[3021]: 4s-query.c:183 couldn't connect to “my_cluster”

[1] https://travis-ci.org/szarnyasg/travis-4s-test
[2] https://api.travis-ci.org/jobs/33973567/log.txt

@swh
Collaborator

OK, so neither the import nor the query were able to connect.

Try running the import with -v -v to see why it couldn't connect, it looks like the backend managed to create the socket ok.

Also you could try telnetting to whatever port your 4s-backnd comes up on (it was 6734 on that last run, but it can be other numbers) to rule out firewalls or selinux or anything like that.

@swh
Collaborator

Oh, just seen that you had one level of --verbose, so you should have seen something... is it possible that stderr is being discarded?

@szarnyasg

@swh I added the -v -v flags and redirected stderr to stdout (I don't think it was discarded, but tried this anyway). Unfortunately, no difference in the output.

However, telnet is able to connect: https://travis-ci.org/szarnyasg/travis-4s-test/builds/33976445

@swh
Collaborator

OK, if telnet is able to connect, it's probably a Multicast issue.

Can you try running avahi-browse _4store._tcp - that should show 4store's multicast adverts.

@swh
Collaborator

OK, so check the you have an MDNS / Multicast server running, I can't remember exactly what it's called on linux, maybe Avahi Server?

@swh
Collaborator

There's tools in the avahi suite that let you test multicast adverts - avahi-browse to read, can't remember how you write them offhand.

@szarnyasg

Well, avahi-daemon is running, but avahi-browse --all shows no results.

+ps auxw
+grep -i 'avah[i]'
avahi     2916  0.0  0.0  32244  1740 ?        S    14:14   0:00 avahi-daemon: running [testing-worker-linux-6-1-30039-linux-3-33978532.local]
avahi     2917  0.0  0.0  32128   556 ?        S    14:14   0:00 avahi-daemon: chroot helper
+service avahi-daemon status
avahi-daemon start/running, process 2916
+avahi-browse --all --resolve --terminate
@swh
Collaborator

Try deliberately creating something with avahi-publish, if that doesn't show up then something is broken in your avahi / fns / multicast stack. e.g. http://wiki.ros.org/zeroconf_avahi/Tutorials/Debugging%20with%20Avahi%20Command%20Line%20Tools

@szarnyasg

Your guess is correct: the stack is broken somewhere.

+sleep 5
+avahi-publish -s DudeMaster _ros-master._tcp 8889
Established under name 'DudeMaster'
+avahi-browse --all --terminate
+grep -i dude

(Interestingly, the sleep and the avahi-publish commands are shown in reverse order -- https://github.com/szarnyasg/travis-4s-test/blob/48e341aedbc3cee1d7a03559d12929a45ec8e3c3/run.sh).

Thanks for the help, I will file an issue to the travis-ci repository.

@swh
Collaborator

OK, great, hopefully it's a simple issue...

One thing I never checked - I think some cloud systems (e.g. AWS) turn off multicast in the kernel - if that's the problem there's an alternative discovery mechanism which doesn't depend on multicast to discover clusters (see man 4s-boss).

@ianmillard

Since @davechallis did his excellent work on 4s-boss and 4s-admin I've stayed as far away as possible from avahi/MDNS (in both clustered and single node deployments)

I've found it to be much more reliable than avahi-daemon, and easier too

Just create /etc/4store.conf eg

[4s-boss]
discovery = sole
nodes = localhost
#nodes=node1;node2;node3;node4

[my_repository]
port = 7890

and then your command sequence would be

4s-boss
4s-admin create-store --segments=2 my_repository
#4s-admin list-stores
4s-admin start-stores my_repository
4s-import -v my_repository foo.ttl
#4s-httpd my_repository
4s-query my_repository "SELECT ..."

(just seen Steve's reply, I'll shut up now...)

@szarnyasg

@swh, @ianmillard thank you both for your help, I finally managed to get it working: https://travis-ci.org/szarnyasg/travis-4s-test/builds/33986661.

The final trick was that I had to change nodes = localhost to nodes = 127.0.0.1 to enforce IPv4. Without that, localhost was resolved to ::1 based on the /etc/hosts file.

I posted an issue to travis-ci about the problems with Avahi: travis-ci/travis-ci#2739.

@szarnyasg szarnyasg referenced this issue in travis-ci/travis-ci Sep 1, 2014
Closed

avahi-browse is not working #2739

@mwjames

After some trial and error setup the following [0] allowed use to run 4store on Travis-CI.

[0] https://github.com/mwjames/SemanticMediaWiki/blob/4store/build/travis/install-services.sh#L67-L115

@mwjames mwjames closed this Jun 17, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment