How to Easily Test Data on Already Trained Data

[amandaloh16]

I already trained my data on part of training and part of a withheld set. so I have a pth.tar file.

I am still confused as to what the terminal command I should use to write the labels for the test data I want to analyze.

I tried this command: python partition/write_Semantic3d.py --SEMA3D_PATH /media/amanda/Seagate\ Expansion\ Drive//Semantic3D_13 --odir "results/custom12/validation_best" --db_test_name testred

with my own custom classes (in same format as Semantic3D) and it complains that it doesn't have files:
(unable to open file: name = './results/custom12/validation_best/predictions_testred.h5')

What am I missing to just run test data on my trained NN? The documentation is a little unclear. Do I have to repartition, create superpoints, again for newly added test data?

[loicland]

Yes you need to compute the partition, the spg and the inference on your target data set. write_semantic3d only convert the super point classification into a .labels file containing the labels per point, as required by semantic3d.com.

If you just want to see the results use the vizualize function (which requires you to perform partition and inference as well).

[amandaloh16]

Do I have to run training as well again? I want to avoid it retraining again. You just load the pth.tar file correct? and with the --test test_reduced flag right?

[loicland]

1 - compute the partition of the train and test sets
2 - train on the train set
3 - inference on the test set
4 - call to write or visualize on the wanted files

[amandaloh16]

Great and just to clarify, inference on test set is just the terminal command for training but with the -----resume RESUME argument correct?

UPDATE:

Right now I am running these commands for a 12 class dataset:
python partition/partition.py --dataset custom_dataset --ROOT_PATH /home/amanda/Semantic3D_small --voxel_width 0.05 --reg_strength 0.8 --ver_batch 5000000

python learning/custom_dataset.py --CUSTOM_SET_PATH /home/amanda/Semantic3D_small

CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset custom_dataset --CUSTOM_SET_PATH /home/amanda/Semantic3D_small/ --epochs 500 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10_0,f_12' --ptn_nfeat_stn 8 --pc_attribs xyzelpsv --nworkers 2 --odir "results/simple/trainval_best"

CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset custom_dataset --CUSTOM_SET_PATH /home/amanda/Semantic3D_small --db_test_name testred --db_train_name train --epochs -1 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10_0,f_12' --ptn_nfeat_stn 8 --pc_attribs xyzelpsv --nworkers 2 --odir "results/simple/trainval_best" --resume "results/simple/trainval_best/model.pth.tar"

I am having difficulty getting the last command to run due to this error:
Traceback (most recent call last):
File "learning/main.py", line 442, in
main()
File "learning/main.py", line 366, in main
acc_test, oacc_test, avg_iou_test, per_class_iou_test, predictions_test, avg_acc_test, confusion_matrix = eval_final()
File "learning/main.py", line 326, in eval_final
confusion_matrix.count_predicted_batch(tvec_cpu, np.argmax(o_cpu,1))
File "/home/amanda/superpointNew/superpoint_graph/learning/metrics.py", line 19, in count_predicted_batch
self.confusion_matrix[:,predicted[i]] += ground_truth_vec[i,:]
ValueError: operands could not be broadcast together with shapes (12,) (8,) (12,)

Not sure how to resolve. changed partition.py, provider.py, main.py in learned, etc to accommodate 12 classes but not sure if i missed somewhere.

[loicland]

Great and just to clarify, inference on test set is just the terminal command for training but with the -----resume RESUME argument correct?

correct!

Not sure how to resolve. changed partition.py, provider.py, main.py in learned, etc to accommodate 12 classes but not sure if i missed somewhere.

Did you adapt the classes field in the return of the get_info function in custom_dataset.py?

If so, can you run the code in debug with a stoppoint at `learning/metrics.py", line 19 and provide here the size of the following variables:

self.confusion_matrix and ground_truth_vec and predicted[i]

loic

[amandaloh16]

Yes this is what I wrote:
`edge_feats = 0
for attrib in args.edge_attribs.split(','):
a = attrib.split('/')[0]
if a in ['delta_avg', 'delta_std', 'xyz']:
edge_feats += 3
else:
edge_feats += 1

return {
    'node_feats': 11 if args.pc_attribs=='' else len(args.pc_attribs),
    'edge_feats': edge_feats,
    'classes': 12, #CHANGE TO YOUR NUMBER OF CLASS
    'inv_class_map': {0:'terrain_man', 
                      1:'terrain_nature', 
                      2:'veget_hi',
                      3:'veget_low',
                      4:'building', 
                      5:'scape',
                      6:'other',
                      7:'vehicles',
                      8: 'traffic_sign',
                      9: 'traffic_light',
                      10: 'pole',
                      11: "guard rail"},`

My other attributes are thus:
confusion matrix
(12, 12)
ground truth vec
(1663, 8)
predicted
()

Trying to debug why ground truth vec is wrong.

[loicland]

Indeed the problem seems to be in the ground_truth_vec.

Some leads:

• did you change the number of labels line 44 of `partition/partition.py?
• the last argument of libply_c.prune (around line 140) is the number of class. Did you change it? If so, what is the shape of the returned labels variable?
• check the size of the field labels in a file in features
• same in a file of superpoint_graphs

[amandaloh16]

1. yes
2. the number class is 0 for test data and 12 for training data
   3.features for test folders is nothing: they are correct for train, assuming because labels are made until inference step for test?
3. same situation in the superpoint graph area for labels->training has values but test data no

[loicland]

There seems to be a problem when building the ground truth label matrix for your test data. Since you don't have labels for your test data, you could simply comment the call to
confusion_matrix.count_predicted_batch(tvec_cpu, np.argmax(o_cpu,1)) in eval_final, since it won't give any meaningful metric anyway (all gt labels are 0).

I will look into it more in-depth soon.

[amandaloh16]

Yep, now I can run the whole pipeline, seem the issue is when you want to do inference on custom test data labels that line trips up in terms of indexing: let me know if you figure it out.

[xvjiabin]

Great and just to clarify, inference on test set is just the terminal command for training but with the -----resume RESUME argument correct?

UPDATE:

Right now I am running these commands for a 12 class dataset:
python partition/partition.py --dataset custom_dataset --ROOT_PATH /home/amanda/Semantic3D_small --voxel_width 0.05 --reg_strength 0.8 --ver_batch 5000000

python learning/custom_dataset.py --CUSTOM_SET_PATH /home/amanda/Semantic3D_small

CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset custom_dataset --CUSTOM_SET_PATH /home/amanda/Semantic3D_small/ --epochs 500 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10_0,f_12' --ptn_nfeat_stn 8 --pc_attribs xyzelpsv --nworkers 2 --odir "results/simple/trainval_best"

CUDA_VISIBLE_DEVICES=0 python learning/main.py --dataset custom_dataset --CUSTOM_SET_PATH /home/amanda/Semantic3D_small --db_test_name testred --db_train_name train --epochs -1 --lr_steps '[350, 400, 450]' --test_nth_epoch 100 --model_config 'gru_10_0,f_12' --ptn_nfeat_stn 8 --pc_attribs xyzelpsv --nworkers 2 --odir "results/simple/trainval_best" --resume "results/simple/trainval_best/model.pth.tar"

I am having difficulty getting the last command to run due to this error:
Traceback (most recent call last):
File "learning/main.py", line 442, in
main()
File "learning/main.py", line 366, in main
acc_test, oacc_test, avg_iou_test, per_class_iou_test, predictions_test, avg_acc_test, confusion_matrix = eval_final()
File "learning/main.py", line 326, in eval_final
confusion_matrix.count_predicted_batch(tvec_cpu, np.argmax(o_cpu,1))
File "/home/amanda/superpointNew/superpoint_graph/learning/metrics.py", line 19, in count_predicted_batch
self.confusion_matrix[:,predicted[i]] += ground_truth_vec[i,:]
ValueError: operands could not be broadcast together with shapes (12,) (8,) (12,)

Not sure how to resolve. changed partition.py, provider.py, main.py in learned, etc to accommodate 12 classes but not sure if i missed somewhere.

i have met the same problom, u can check /learning/spg.py. In the line 73
node_gt_size = np.concatenate([f['sp_point_count'][:].astype(np.int64), np.zeros((N,8), dtype=np.int64)], 1)
u can try change 8 of np.zeros((N, 8) to 12(your classes), like np.zeros((N,12).
i hope this is useful for you.