Matching two Meshes

[roaasoloh]

Previously and upon you assist me in this: #83 (comment)
now what I have to ask, if I have two meshes, and I need to make matching between them, we already have interest points, and from these points, and after applying few strategies, I got matching between meshes,
So now, instead of having:
a = [1,4,7,2,9,10,50,33,25]
I got a new array having for example: b = [1,50,33,25] and this the common points between the two meshes
My question would it be possible to show the two meshes, showing the main interest points, as well make lines that refer to matching depending on b
like the image obtained by someone, and the color of the line will be later, depending on whether this match is correct or not.
Would it be possible to do this using this library, please?

[marcomusy]

I'm not sure what a and b mean in you example, are these the indices of some mesh points?

In any case these examples might be useful for you:

• aligning:
  https://github.com/marcomusy/vtkplotter-examples/blob/master/vtkplotter_examples/basic/align1.py
  https://github.com/marcomusy/vtkplotter-examples/blob/master/vtkplotter_examples/basic/align2.py

• measuring mesh distances:
  https://github.com/marcomusy/vtkplotter-examples/blob/master/vtkplotter_examples/basic/distance2mesh.py

You can use Line() to link specific points.

[roaasoloh]

Yes Sir, a and b are indices for mesh points,

So I have two meshes I should show them in front each other, like in the image attached before, as well as I mentioned in the mean question, a is the interest points we showed firstly (white dots) and now b is the matched dots that we need to link between the two meshes( through a line) showing that this mesh matches that mesh in the points indicated in b surely we have the 3d coordinates for them!

Hopefully, my ideas get clearer!

In fact, the links you gave me regarding aligning, it doesn't benefit me, and regarding measuring distances I also believe no need for it, I'm trying to obtain Euclidean distance in order to obtain the match

giving example would be like this:
I have Mesh a and Mesh b and have the interest points for them like a_ip and b_ip now the more thing that we want to visualize is this:

[[[ 0.          0.          0.        ]
  [-0.0028908  -0.00196051 -0.00350424]]

 [[-0.03966275  0.06078086 -0.02791364]
  [-0.03427645  0.05252666 -0.0241229 ]]

 [[-0.03966275  0.06078086 -0.02791364]
  [-0.03427645  0.05252666 -0.0241229 ]]

 ...

 [[-0.05971977  0.05101534 -0.01131839]
  [-0.02903689  0.03437778 -0.06774345]]

 [[ 0.00041986  0.05901061 -0.05596179]
  [-0.0461167   0.06156679 -0.00700319]]

 [[-0.06774878  0.04416654  0.00857729]
  [ 0.00350887  0.00303647 -0.00171688]]]

Where here we have first one is the coordinates(or we can get it as index) from mesh a and the second is a coordinate for a point from Mesh b each array represents a matching line between two meshes.
Much Appreciated for your assistance.

[marcomusy]

Try:

from vtkplotter import *
import numpy as np

a = load(datadir+'man.vtk')
a_ip = list(range(0, 20)) # pick some point indices
a_pts = a.points()[a_ip] # 3d coords of those points

b = a.clone().pos(1,0.2,0).c('violet')
b_ip = list(range(1000,1020))
b_pts = b.points()[b_ip]

lns = Lines(a_pts, b_pts, lw=2)
scal = mag(b_pts - a_pts)  # length of line as color
lns.cellColors(scal, cmap='jet')
lns.addScalarBar(title='distance')

show(a,b, lns, axes=1)

[roaasoloh]

Dear Sir,
It's me again,
I appreciate if I can get assistance with this:
My matching is done through several ways(I'm trying to get the best), so I will appreciate if there's a possibility to make visualizing using them,
method one:

def mesh_matching(mesh1, mesh2):
    kp1, fv1 = extract_all_fv(mesh1)
    kp2, fv2 = extract_all_fv(mesh2)
    # print("feature 1",fv1)
    # print("feature 2",fv2)
    association = []
    # matching keypoint using feature vector
    for i in range(len(fv1)):
        similarities = [np.linalg.norm(fv1[i]-fv2[j]) for j in range(len(fv2))]
        print(len(similarities))
        best = np.argmax(similarities)
        #print(similarities[best])
        # remove fake association with some thresholding
        if similarities[best] > 0.3:
            association.append(similarities)
        #print(association)
    association = np.array(association)
     return (kp1, kp2, association)

method two:(Favorable)

# Matcher
bf = cv2.BFMatcher(cv2.NORM_L2, crossCheck=False)
matches_bf = bf.match(des1,des2)
print(len(matches_bf))

# FLANN Matcher
FLANN_INDEX_KDTREE = 0
index_params = dict(algorithm = FLANN_INDEX_KDTREE, trees = 5)
search_params = dict(checks=50)   # or pass empty dictionary
flann = cv2.FlannBasedMatcher(index_params,search_params)
matches = flann.knnMatch(des1.astype('float32'),des2.astype('float32'),k=1)
matches = sum( matches, [] )

can I get guidance for visualizing meshes with there key points and the matching depending on this,
for example, when using the second method for images, this could be done through

# Draw Matches
im_matches = cv2.drawMatches(mesh1,kp1,  mesh2,kp2,  matches, None)
cv2.imshow( 'matches', im_matches)

in addition to this, later on, there might be choosing different colors if this is a correct match or not.
Thank you

My result would be something like the images attached:

[marcomusy]

Hi, I'm afraid i'm not familiar with neither cv2 nor flann. If you already have the matching points as coordinates or indices between meshes
(you can then get the coordinates using mesh.points()[mylist_of_indices]),
then you can visualize everything you wish very easily using classes Line DashedLine Point Sphere Glyph etc.
Coloring these objects with scalars and/or transparency should be equally easy by following the above examples. More visualization examples are given here.

[roaasoloh]

Hello Again,
I hope it will be the last problem, to close the issue
Thanks for your support :)
I had a problem regarding the indices of my match,
now the case is as following:

    mesh1 = load("/home/roaa/Desktop/PhD/Models/Apple/apple.ply").normalize()
    mesh2 = load("/home/roaa/Desktop/PhD/Models/Apple/apple_translated.ply").normalize()
    kp1,kp2,matches = mesh_matching(mesh1,mesh2)
    print(matches[:,0],mesh1.points()[matches[:,0]])
    print(matches[:,1])
    lns = Lines(mesh1.points()[matches[:,0]],mesh2.points()[matches[:,1]],lw=2)
    scal = mag(mesh2.points()[matches[:,1]] - mesh1.points()[matches[:,0]])  # length of line as color
    lns.cellColors(scal, cmap='jet')
    lns.addScalarBar(title='distance')
    rpts = Points(mesh1.points()[kp1], r=5, c='white')
    rpts1 = Points(mesh2.points()[kp2], r=5, c='blue')
    show(mesh1,rpts,mesh2.pos(1,0.2,0).c('violet'),rpts1, lns, axes=1)

my results are like this:

so it didn't show separately!! like your visualizing? as well how to ensure that rpts1 showed on mesh2, not on mesh1?
also if I load meshes without .normalize() on one side the matching points differs!! and the visualization is like this:

Here is my case, I have 2 matching points! and if I try the same object it gives full matching, but in visualization, nothing is shown (lines), what I'm doing wrong, please?
also kindly, how I may save the output directly, or after observing it.
and if for example I just want to show lines, since I would like to make the color of lines dependable on wether it is a correct match or not, not dependable on distance scalar.

[marcomusy]

You are shifting the second mesh inside the show(), that deosn't make much sense to me... check out my version of your example:

from vtkplotter import *

mesh1 = load(datadir+'apple.ply').alpha(0.2)
mesh2 = load(datadir+'apple.ply').x(3).c('v').alpha(0.2)

pts1, pts2 = mesh1.points(), mesh2.points()

matches1 = [  0,100,200]
matches2 = [300,400,500]

lns = Lines(pts1[matches1], pts2[matches2], lw=4)
scal = mag(pts1[matches1] - pts2[matches2])
lns.cellColors(scal, cmap='viridis').addScalarBar(title='distance')
rpts1 = Points(pts1[matches1], r=10, c='white')
rpts2 = Points(pts2[matches2], r=10, c='blue')

show(mesh1, mesh2, lns, rpts1, rpts2, axes=1)
screenshot('a.png')

• if you have lines to indicate wrong matches you can use a different Line() object with a fixed color.
• you can save the image with screenshot()

[roaasoloh]

Dear Sir,
Kindly I have two questions if you please.

• 1:
  When you showed me the way of matching by the lines the two meshes, you used:

scal = mag(b_pts - a_pts)

What about if I have for the two meshes two sets good, and bad
good = [(2,4),(10,1),(15,9)...] where the first number is from the first mesh, and the second is for the second, same for bad
I would like to use only two colors showing the matching on the mesh. is it possible?

• 2:
  What about if I have the 3d coordinates but I need to get their indexes? what function could be used if I have several xyz-coordinates and I need to get each for example it's number [3.4 5.1 6.3] coordinates refers to [4] point!!

[marcomusy]

1. Then you don't need a scalarbar. Try

from vedo import *
import numpy as np

mesh1 = load(datadir+'apple.ply').alpha(0.2)
mesh2 = mesh1.clone().x(3).c('v')

pts1, pts2 = mesh1.points(), mesh2.points()

good_matches = [  0,100,200]
bad_matches  = [300,400,500]

good_lns = Lines(pts1[good_matches], pts2[good_matches], lw=3, c='g')
bad_lns  = Lines(pts1[bad_matches],  pts2[bad_matches], lw=4, c='r')
show(mesh1, mesh2, good_lns, bad_lns, axes=1)#.screenshot('a.png')

2. possible solution:

########################## find index from coordinate
#print(pts1[good_matches])
pp = [[ 1.03942804e-01,  1.83283873e-02, -9.15606380e-01],
      [ 0, -3.48307818e-01, -9.15e-01],
      [-6.23666346e-01, -1.09974377e-01, -9.41993356e-01]]

def find_index(array, p):
    array = np.asarray(array)
    if not utils.isSequence(p): p = [p]
    idxs = [np.sum((array-q)**2, axis=1).argmin() for q in p]
    return idxs

print(find_index(pts1, pp))

[roaasoloh]

Sorry, I'm re-opening this issue,
But in fact, I still have a problem please, and I believe most probably it's from the visualizing,
In the image below, it shows the key points on the objects, at the same time, I Have good and bad matches, but the problem is having key points not linked by lines?

My code is as follows:

mesh1 = load("/home/roaa/Desktop/PhD/Models/OwnDatabase/Ours/chair-poses/"+names[3]).normalize().c('blue')
    # #print(mesh1.points())
mesh2 = load("/home/roaa/Desktop/PhD/Models/OwnDatabase/Ours/chair-poses/"+names[6]).normalize().x(3).c('green')
good_lns = Lines(mesh1.points()[good[:,0]], mesh2.points()[good[:,1]], lw=2,c = 'g')
bad_lns = Lines(mesh1.points()[bad[:,0]], mesh2.points()[bad[:,1]], lw=2,c ='b')
rpts = Points(mesh1.points()[keypts[3]], r=4, c='black')
rpts1 = Points(mesh2.points()[keypts[6]], r=4, c='blue')
show(mesh1, mesh2, good_lns,bad_lns, rpts, rpts1, axes=0)

[marcomusy]

I cannot detect any mistake in the above code, I suggest you double check the indices you use to select array slices.

[roaasoloh]

So I tried on simpler models, the previous lines were taken dependable on files, now I only tried on two models, as in the picture, it is shown on the tails there are no matches!

the code for this result is directly like this:

mesh1 = load("/home/roaa/Desktop/PhD/Models/OwnDatabase/Ours/dolphin-poses/m13.obj").normalize().c('blue')
#     # #print(mesh1.points())
mesh2 = load("/home/roaa/Desktop/PhD/Models/OwnDatabase/Ours/dolphin-poses/m14.obj").normalize().x(3).c('green')
from  FeatureVector_cluster import extract_all_fv
keypoints,descriptor = extract_all_fv("/home/roaa/Desktop/PhD/Models/OwnDatabase/Ours/dolphin-poses/m13.obj")
keypoints1,descriptor1 = extract_all_fv("/home/roaa/Desktop/PhD/Models/OwnDatabase/Ours/dolphin-poses/m14.obj")
distance_matrix= cosine_similarity(descriptor,descriptor1)
rows,cols = linear_sum_assignment(distance_matrix)
mat = list(zip(rows,cols))
good_match = 0
bad_match = 0
good = []
bad = []
for r,c in mat:
    if distance_matrix[r][c] >= 0.8:
        good.append([r,c])
        good_match += 1
    else:
        bad.append([r, c])
        bad_match += 1
good = np.array(good)
print("Number of good matches is: ",len(good))
bad = np.array(bad)
print("Number of bad matches is: ",len(bad))
good_lns = Lines(mesh1.points()[good[:,0]], mesh2.points()[good[:,1]], lw=2,c = 'g')
bad_lns = Lines(mesh1.points()[bad[:,0]], mesh2.points()[bad[:,1]], lw=2,c ='b')
rpts = Points(mesh1.points()[keypoints], r=4, c='black')
rpts1 = Points(mesh2.points()[keypoints1], r=4, c='blue')
show(mesh1, mesh2, good_lns,bad_lns, rpts, rpts1, axes=0)

[marcomusy]

I have no way to reproduce your plot - and it's not clear what you are expecting to see "on the tails".

Make sure you have the latest version of the code:
pip uninstall vedo
pip install -U vedo

then check what indices/points correspond to in arrays mesh1.points()[good[:,0]] etc, because the problem may be there.

[roaasoloh]

So I did what you said,

What I'm expecting on tails is finding lines between the points there!

I mean assuming I have 30 key points which are visualized on each dolphin! there's 22 good, and 8 bad
So there must be lines connecting the 30 key points either black for good, or blue for bad!

Did you get my point?

And regarding the indices I didn't find any problem with them, I even replace NumPy and use this:

good.append([r,c])
....
...
good_1 = [row[0] for row in good]
good_2 = [row[1] for row in good]
....
good_lns = Lines(mesh1.points()[good_1], mesh2.points()[good_2], lw=2,c = 'g')

[marcomusy]

No, I don't get your point, that's very confused.

You should find a reproducible example otherwise I don't know how to help.

[roaasoloh]

Please could you clarify what you mean by reproducible example? All the problem is: The keypoints that I extracted and show them on the model should be linked with lines from the first model to the second, while this is not appearing! Like in the tail of the dolphin, there are keypoint that appears but don’t have links with that on the other dolphin?

…

Sent from my iPhone

On 15/10/2020, at 10:28 PM, Marco Musy ***@***.***> wrote:  No, I don't get your point, that's very confused. You should find a reproducible example otherwise I don't know how to help. — You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub, or unsubscribe.

[marcomusy]

well it's very hard to see anything about the links between the two dolphins.

reproducible example means something I can run which exhibits and unexpected or undesired behavior.

[roaasoloh]

Sorry for reaching again, although I wasn't able to configure how I may make a reproducible example, but the idea despite dolphin or any model I'm trying to match.
I'm having a set of key points! these keypoints are visualized on the two models, and I performed a match between them using your suggested code regarding good and bad matches, the summation for them is equal to that number of key points, so definitely all points on the model should be matched to the second one!

That's my point!

Thank you

[marcomusy]

You need to learn how to debug your code!
the 2 dolphin pic above is way too confused, I can't see anything from there.
Print a (small!) list of points that you expect to see linked in the picture - and check if indeed they are joined by a line.

[roaasoloh]

Hello,
Please I was wondering about visualizing points in a specific way if it's possible:
I have the following list of indices of vertices:

[[  0   1 165 178 191 239 238 254 255 240]
 [ 10   9  11 168 180 193 243 242 256 127]
 [ 20 163  21 174 175 187 195 200 209 220]
 [ 30  49  31 181 194 245 244 257 128 114]
 [ 40 186 185 252 276 273 274 146 142 145]
 [ 50  55  77  66  56  51  52  65  67  73]
 [ 60  59  71  80 136 135 268 222 213 204]
 [ 70  58  79  85  84 134 228 227 223 214]
 [ 80 136 135 268 222 213 204 203 207  62]
 [ 90  91  43  50  55  77  66  56  51  52]
 [100  99  48  29  11 168   9  10 154 155]
 [110  94  93  45  26  25 177 190 237 236]
 [120 119 124 153 152 123 122 121 110  94]
 [130 117 260 250 251 107 108 106 116 129]
 [140 159 141 139 160 161 147 109  39  41]
 [150 151 278 269 137 138 148 149 277 207]
 [160 161 140 159 141 139 162 145 146 142]
 [170 179 192 241 240 255 254 238 239 191]
 [180 193 243 242 256 127 113 100  99  48]
 [190 177  25  26  45  93  94 110 121 124]
 [200 209 220 201 196 195 197 210 208 216]
 [210 208 216 219 220 209 200 195 197 173]
 [220 209 200 195 197 210 208 216 219 201]
 [230 229 226 131  83  74  78  88  55  77]
 [240 241 192 179 170  12  28  47  97  98]
 [250 251 107 108 117 130 260 248 249 183]
 [260 130 117 108 107 251 250 248 249 183]
 [270 283 271 281 282 276 252 185 186  40]
 [280 267 265 264 279 266 263 262 253 236]]

Where I would like to visualize each vertex of the first column different than the others,
In another world, in each list, I have the first index refers to the main vertex, and the rest of the list is the neighborhood for this vertex, so I would like to differentiate between them in visualizing.
[ 10 9 11 168 180 193 243 242 256 127] in this list for example, 10 is the desired vertex, and the others are its neighbors.

Thank you for your support.

[marcomusy]

you can extract the coordinates with

import numpy as np

mylist=[[  0,   1, 165, 178, 191, 239, 238, 254, 255, 240],
        [ 10,   9,  11, 168, 180, 193, 243, 242, 256, 127]]
mylist = np.array(mylist)

mycenters = np.array(mylist)[:,0]
myneigh = np.delete(np.array(mylist), 0, 1).ravel()

print(mycenters)
print(myneigh)

# coords:
print(mymesh.points()[mycenters])
print(mymesh.points()[myneigh])

(please understand that I only have time resources for helping with vedo not with general questions about numpy)