Evaluation Client

The evaluation client calls the ROS service 'group_classification' for each scene in a supplied JSON scenes file(see here). The perception for each scene is read from a perceptions file (Simulated Perception).

The output of the classification is stored to a flat text file for later processing.

Evaluation client usage

Usage: rosrun strands_qsr_evalution evaluation_client.py [options]

Options:
  -h, --help            show this help message and exit
  -p FILE, --perception_filename=FILE
                        read perceptions from FILE
  -s FILE, --scenes=FILE
                        load the scenes from FILE
  -o FILE, --output=FILE
                        store results in FILE
  -O, --overwrite       overwrite output file if already exists.
chris@heatwave:~/documents/strands/strands_qsr.wiki$ 

for example

rosrun strands_qsr_evalution evaluation_client.py -s data/simulation/bham_office_desk_500.json -p simulated_perceptions.json -o run1_results.txt

All object classes that exist in the specified simulated perceptions file will be considered - ie. to test on a subset of the object classes in a given scenes file, create a simulated perception file with only the desired object types.

Output format

The results file looks like this:

--
scene1
Pencil	Monitor	Cup	Telephone	PC	Keyboard	
Pencil	Laptop	Cup	Mouse	PC	Keyboard	
Pencil	Cup	Cup	Mouse	PC	Keyboard	
--
scene2
Pencil	Monitor	Monitor	Telephone	Bottle	Keyboard	Pencil	
Pencil	Monitor	Monitor	Telephone	Bottle	Cup	Stapler	
Pencil	Monitor	Monitor	Telephone	Bottle	Cup	Stapler	
--
...

each labelled scene is specified by 5 lines:

1. A separator '--'
2. The scene id as read from the scenes file
3. A tab separated list of ground truth labels
4. A tab separated list of perception labels
5. A tab separated list of relation+perception labels

So, if perception is 100% then lines 3 & 4 will be the same.