No Values received on MQTT

[eifinger]

Using Docker when subscribing to the mqtt topics on the Find3 mosquitto server using mosquitto_sub on the same server no messages are received.

I can can confirm that the mosquitto server is running I got a "Connected not authorized" when connecting without user and password.

[schollz]

I can't replicate this. Can you give me more details?

[eifinger]

docker-compose.yaml:
find3server:
container_name: find3server
restart: unless-stopped
image: schollz/find3
volumes:
- /home/eifinger/find3server:/data
ports:
- "11883:1883"
- "8003:8003"

Added one phone with family name "eifinger". Added two rooms.
http://hostname:8003 shows the dashboard and the location of my phone correctly

Got the mqtt password with
curl http://localhost:8003/api/v1/mqtt/eifinger`

Executed
mosquitto_sub -t "eifinger/location/#" -v -p 11883 -u eifinger -P xxxxx
mosquitto_sub -t "#" -v -p 11883 -u eifinger -P xxxxx

Both showed no messages

[eifinger]

I investigated a bit further. I added the env variables for mqtt to the image and recreated the container.
I see the exact same behaviour as in #70

[schollz]

@eifinger Thanks, I think the error is that mosquitto still is not populating the right configuration. Looking into it now. Mosquitto really is the bane of my existence :(

[schollz]

@eifinger Ok, seems to work for me now. Open it back up if it doesn't

[eifinger]

Thanks for being so fast on this.
I can confirm that the authorization now works. I can only connect using admin and created family credentials.

However there are still no messages whatsoever.
Just to be sure, if I see a popup on the dashboard with the current location of my phone I should see a mqtt message too?

[schollz]

@eifinger That's correct. Can you try with the testdb?

$ cd $GOPATH/src/github.com/schollz/find3/server/main/testing
$ python3 submit_jsons.py http://localhost:8005 testdb.learn.1439597065993.jsons

Then create MQTT credentials for testdb, and do mosquitto_sub for testdb and then run the submit_jsons again.

[eifinger]

Here is what I did:
sudo docker exec -it find3server /bin/bash
cd main/testing
pip3 install requests
python3 submit_jsons.py http://localhost:8005 testdb.learn.1439597065993.jsons
exit
curl http://localhost:8003/api/v1/mqtt/testdb {"message":"problem giving HUP: exit status 1","success":false}
Since this didn't work I did a mosquitto_sub with the admin credentials and then inside the docker container resubmitted the json file. I still didn't see any messages.

[eifinger]

I tried again, without messing with the docker container, thus only
python3 submit_jsons.py http://localhost:8003 testdb.learn.1439597065993.jsons
curl http://localhost:8003/api/v1/mqtt/testdb
mosquitto_sub -t "#" -v -p 11883 -u testdb -P xxxx
python3 submit_jsons.py http://localhost:8003 testdb.learn.1439597065993.jsons

No messages

[schollz]

@eifinger I'm not sure about your test - you're running the test within the Docker container? That's not exactly what I meant. Here's what I do (which doesn't replicate your error):

Starting a brand new instance:

docker pull schollz/find3
sudo rm -rf ~/FIND_DATA && mkdir ~/FIND_DATA
docker run -p 11883:1883 -p 8015:8003 -v /home/$USER/FIND_DATA:/data -e MQTT_ADMIN=ADMIN -e MQTT_PASS=PASSWORD -e MQTT_SERVER='localhost:1883' -d -t schollz/find3
go get github.com/schollz/find3
cd $GOPATH/src/github.com/schollz/find3/server/main/testing
python3 submit_jsons.py http://localhost:8015 testdb.learn.1439597065993.jsons

Now to test it:

In one terminal:

http GET localhost:8015/api/v1/mqtt/testdb
mosquitto_sub -h 127.0.0.1 -p 11883 -u testdb -P XXX -t '#'

And then in another terminal:

cd $GOPATH/src/github.com/schollz/find3/server/main/testing
python3 submit_jsons.py http://localhost:8015 testdb.learn.1439597065993.jsons

after which the first terminal will spit out lots of results.

I suggest you try this - instead of erasing your FIND_DATA you could just replace that with a different folder. If this doesn't work, it means there is a weird bug. I suspect that this will work though, and that maybe the reason yours doesn't work is something that is not updating in the mosquitto database. Its not a problem that I've had so I can't really tell you more without eliminating this possibility first.

[eifinger]

I did exactly those steps and can see no messages.
Only difference is that I used wget to get the python script and jsons file.

[eifinger]

I just tried this on my Win10 machine (docker running linux containers) and got the same result: No mqtt messages

[schollz]

@eifinger Can you try building the Docker container yourself?

cd $GOPATH/src/github.com/schollz/find3 && git pull
docker build -t find3

And then use find3? I can only imagine that this is a Docker problem now because that should be the only difference between our setups now. Can you also print the results of docker version?

[eifinger]

I just built the docker image and followed the test procedure with sadly still no mqtt messages.
Output of docker version:

Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:17:20 2018
OS/Arch: linux/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:15:30 2018
OS/Arch: linux/amd64
Experimental: false

[schollz]

@eifinger You have a newer version of Docker than I do. I upgraded Docker, and rebuilt and re-tested and it still works just fine for me. Are you sure the Docker image you built didn't use any of the cached things? Can you try docker build --no-cache -t find3?

[eifinger]

@schollz I just did that. Still no messages. However I found out that no messages are received whatsoever. Even if I publish them myself
I don't get any when connected with
mosquitto_sub -h 127.0.0.1 -p 11883 -u testdb -P 4v72x -t "#" -v -q 2

mosquitto_pub -h 127.0.0.1 -p 11883 -u testdb -P 4v72x -t "/test" -m "test"

[eifinger]

This is really annoying that it behaves completely different on our two systems. Thank you for the help and communcation. I am happy to help if you have an idea how =)

[seaverd]

schollz,

You referred me to this thread...here is what I get for docker version and when trying to query for my password.

root@Tower:~# docker version
Client:
 Version:      17.09.1-ce
 API version:  1.32
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:21:47 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.09.1-ce
 API version:  1.32 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   19e2cf6
 Built:        Thu Dec  7 22:28:28 2017
 OS/Arch:      linux/amd64
 Experimental: false
root@Tower:~# curl http://localhost:8005/api/v1/mqtt/seaverd
{"message":"problem giving HUP: exit status 1","success":false}root@Tower:~#

[seaverd]

When I try and do the following withing the docker:

mosquitto_sub -h 127.0.0.1 -p 11882 -u MYUSERNAME_PASSED AT DOCKER RUN -P MYPASSWORD PASSED AT DOCKER RUN -t '#'

I get connection refused....its like the MQTT_Pass sent at docker run is not working.

[eifinger]

@seaverd Are you sure you have the latest version of find3? This looks like an issue before the last commit.
You can try docker pull scholzz/find3 and create a new container.

[seaverd]

I am sure...just pulled into a new container....same thing.

[schollz]

@seaverd Also make sure to clear your previous find data. Try doing the steps exactly in #90 (comment) and then try inserting fingerprints while connected to the mosquitto as admin

[schollz]

@eifinger Could you share your log? (~/FIND_DATA/logs/main.stdout)

[eifinger]

You can find it under: https://gist.github.com/eifinger/20072c071cd3696d5f6471c0ce03de82

[schollz]

@eifinger Thanks. You are getting a Network Error : dial tcp 127.0.0.1:1883: connect: connection refused inside the Docker container (https://gist.github.com/eifinger/20072c071cd3696d5f6471c0ce03de82#file-main-stdout-L11). That shouldn't happen.

That is the mosquitto port, which should bind to 11883 on your host. Is 11883 on your host open? Maybe you could try changing that port to 11884 or something. Otherwise I don't really know why an internal port inside the Docker would be refused

[eifinger]

Inside the Docker container: Are you sure you start the mosquitto server before the rest?
I attached to the Container and can connect to the broker with mosquitto_sub
EDIT: I tried with 11884 as Mqtt port and checked that's it's free before. Same entry in the logfile

[schollz]

@eifinger That's a good idea. I added a time.Sleep when starting mqtt. That should fix this if it is indeed some kind of race condition for mosquitto to start. 8a452a2

You could rebuild the Docker container and try again

[eifinger]

Works!
With my locally built version. I will now set it up and give it a complete test run.

[eifinger]

I can only real world test it when I am home in my wifi again.
I think this issue is resolved. If not I can reopen it again. Thanks for your help!

[schollz]

@eifinger Thanks for your patience! Let me know if you run into problems again

[seaverd]

I didn't delete my find data...but I see the two of you found the solution. I pulled the updated docker and can confirm that my issue is resolved as well. Great work!

[seaverd]

So I am still having an issue....all works fine. However if I stop docker then restart it, i can't connect to mqtt. If I remove container and delete find data and pull new container it works...until I restart.

[eifinger]

@schollz I just got the chance to actually start using find3 and ran into the mqtt issue again. I'm on the latest docker version (love the added ui element to generate the mqtt password) with a clean setup.

In main.stdout I see the following line:
Network Error : dial tcp: address tcp/1883': unknown port

This is my docker-compose.yaml:

find3server:
  container_name: find3server
  restart: unless-stopped
  image: schollz/find3
  environment:
    - MQTT_ADMIN=admin
    - MQTT_PASS=mypass
    - MQTT_SERVER='localhost:1883'
  volumes:
    - /home/eifinger/find3server:/data
  ports:
    - "11883:1883"
    - "8003:8003"

[schollz]

@eifinger what if you don't use docker-compose?

[eifinger]

Works when I don't use docker-compose. What the fuck?!

[schollz]

Yeah @eifinger I was looking into it and I'm totally perplexed. Thought it might be related to docker/compose#4799 but that didn't seem to help at all. I'd be interested if you find anything out, I'll keep looking too.

[eifinger]

I will definitely do. I suggest we open a new issue, because I think this has nothing to do with your code but with docker-compose port binding

[schollz]

Agreed