Warehouse plugin without custom third-party package

[mikeferguson]

This is a metaticket to cover getting some form of warehouse_ros db plugin working in J/K as part of WMD.

Some background:

• https://github.com/TheBrewCrew/warehouse_ros_mongo/tree/jade-devel is a plugin that might work, but hasn't been touched in a year, and was not really reviewed at that time.
• We could also maybe create a new plugin that uses something simple like SQLite and avoid the whole "mongo has totally different APIs on every released Ubuntu" problem -- although I have not looked at our warehouse API to see how feasible this is.

Other references:

• Can't launch demo.launch with db:=true #75

@alexhenning is currently looking at this

[v4hn]

This resolved part of the issue: #75

I still believe we need a warehouse plugin that relies on a database-client that works out-of-the-box on debian and standard Linux distributions.

[mikeferguson]

During WMD, @alexhenning started on an SQLlite-based database client -- which should be readily available on all platforms. Unfortunately he ran out of time to finish it, we're going to continue and find a time to wrap that up (but I'm not sure exactly when)

[mikeferguson]

I should also mention, I ran out of time to actually get the mongo plugin mentioned above tested/ported, as it took longer than expected to get Kinetic things happy for fully functional simulation of Fetch+MoveIt

[v4hn]

@mikeferguson @alexhenning what's the state of the SQLite based client?
With https://github.com/ros-planning/warehouse_ros_mongo and the build instructions around,
the original problem in this issue is mitigated.
Now, the remaining problem is not actually getting anything to work but instead get something to work that is based on standard libraries packaged in the major linux distributions.
So maybe the title of this issue should be updated?

[k-okada]

@v4hn @mikeferguson
hi, finally, I have time to remove all mongo code from warehouse_ros and use sqlite to store datas, please look at https://github.com/k-okada/warehouse_ros_sqlite and https://github.com/k-okada/warehouse_ros_dummy and hope we can re-enable launch moveit with db:=true options,

[v4hn]

Awesome!

@beatrizleon Shadow is one of the main users of the warehouse that I know of.
Do you think someone could try this sqlite fork with your setup?
I guess you would also be happy to avoid the mongodb too...

[giusebar]

@v4hn @mikeferguson @k-okada
We tested the warehouse_ros_sqlite repository and it works fine. It would be nice to proceed adding this as a default warehouse for moveit.

[gleichdick]

Oh, I just noticed that I'm not the first one implementing a sqlite backend for warehouse 😅
My solution does not use BSON, it stores the metadata in columns for each collection. You can find it here.
There is also some discussion in #1659

[felixvd]

I just tried setting up warehouse - the process sucks balls. The first line in the official doc doesn't work in Melodic. Googling the error leads to unrelated ROS Answers pages, and a workaround which could hardly look dodgier. The only other link in the tutorial leads to a page that hasn't been updated since Groovy. @gleichdick 's graciously detailed Readme (which I appreciate a lot) refers to the same dysfunctional documentation right after the build instructions.

In our Benchmarking tutorials, we simply assume that people have warehouse set up, but it seems that there is no tutorial that sets up a database, or even answers fundamental questions about warehouse:

• How do you install it? Does it come installed?
• How do you start it up?
• Do you need to create a database? How?
• What is supposed to connect to it? How do you know it worked?

Evidently people manage to set it up, but I am not above assuming that if I have difficulties, naturally it is the tools' or documentation's fault. In this case I am actually quite sure. The documentation sucks. Let's fix this. Pinging @captain-yoshi because I know he recently did magic with this somehow.

edit: moveit/warehouse_ros#42

[captain-yoshi]

For Noetic I had to install these:

sudo apt install mongodb-server-core
sudo apt install ros-noetic-warehouse-ros
# Not sure if the command below is needed...
sudo apt install ros-noetic-warehouse-ros-mongo

For debugging purposes you can check some logs from mongod:

mongod -v
# This is normal as we save the db in the moveit_config package
2021-03-12T11:43:57.327-0500 I STORAGE  [initandlisten] exception in initAndListen: NonExistentPath: Data directory /data/db not found., terminating
2021-03-12T11:43:57.327-0500 I CONTROL  [initandlisten] now exiting
2021-03-12T11:43:57.327-0500 I CONTROL  [initandlisten] shutting down with code:100

Finally, you will maybe have this error at one point:

[ERROR] [1615567759.671633]: Mongo process exited with error code 100

That is because you have forgotten to kill your roslaunch XXX_moveit_config demo.launch db:=true. The tutorials says to kill it, but sometimes I forgot when switching between storing scenes and running the benchmark.

In my setup, there is also an infinite recursion with getting distance to collision in this line within FCL. Did not report it yet but maybe because of my setup.

[simonschmeisser]

moveit/moveit_tutorials#478 contains some problems I encountered with the benchmarking tutorial

[felixvd]

With the last line of those install instructions, the instructions in the Benchmarking tutorial work for me. Both @gleichdick 's and @k-okada 's warehouse_ros_sqlite packages worked for me as well, after I entered the full path to the file in the Rviz plugin.

I would be in favor of using it as the default plugin, although in terms of setup both seem roughly equivalent (terrible without documentation, short with). The benchmarking tutorial and this benchmarking package use the mongodb settings as the default, but that should be trivial to change.