Question about using mesh.specialSets['MinI_VertexSet'] as a boundary index with parallel run

[tyszwh]

Hi, @julesghub
I'm having a problem with mesh.

I want to use mesh.specialSets['MinI_VertexSet'] as the boundary index to assign a value to. And if it is running in a single thread, there is no problem, I can get the correct boundary index and value. As shown in the guide example.

import underworld as uw
mesh = uw.mesh.FeMesh_Cartesian(elementRes=(8,8))
print("Available special sets for this mesh are:")
print(mesh.specialSets.keys())
leftset = mesh.specialSets['MinI_VertexSet']
rightset = mesh.specialSets['MaxI_VertexSet']

print("\nIndices in the left set wall set:")
print(leftset)
print("\nIndices in the right set wall set:")
print(rightset)

Available special sets for this mesh are:	Global element size: 8x8
	Local offset of rank 0: 0x0
	Local range of rank 0: 8x8

dict_keys(['MinI_VertexSet', 'Left_VertexSet', 'MaxI_VertexSet', 'Right_VertexSet', 'MinJ_VertexSet', 'Bottom_VertexSet', 'MaxJ_VertexSet', 'Top_VertexSet', 'AllWalls_VertexSet', 'Empty'])

Indices in the left set wall set:
FeMesh_IndexSet([ 0,  9, 18, 27, 36, 45, 54, 63, 72])

Indices in the right set wall set:
FeMesh_IndexSet([ 8, 17, 26, 35, 44, 53, 62, 71, 80])

But If it's running in parallel, I can't get all the indexes, in some cases they are Null values, depending on the number of threads in parallel (see the result of the same example). This prevents me from calling these boundary indexes correctly.
How can I solve this problem?

mpirun -np 4 python3 mesh.py

        Global element size: 8x8
        Local offset of rank 0: 0x0
        Local range of rank 0: 4x4
Available special sets for this mesh are:
dict_keys(['MinI_VertexSet', 'Left_VertexSet', 'MaxI_VertexSet', 'Right_VertexSet', 'MinJ_VertexSet', 'Bottom_VertexSet', 'MaxJ_VertexSet', 'Top_VertexSet', 'AllWalls_VertexSet', 'Empty'])

Indices in the left set wall set:
FeMesh_IndexSet([ 0,  5, 10, 15, 20, 30])

Indices in the right set wall set:
FeMesh_IndexSet([])
Available special sets for this mesh are:
dict_keys(['MinI_VertexSet', 'Left_VertexSet', 'MaxI_VertexSet', 'Right_VertexSet', 'MinJ_VertexSet', 'Bottom_VertexSet', 'MaxJ_VertexSet', 'Top_VertexSet', 'AllWalls_VertexSet', 'Empty'])

Indices in the left set wall set:
FeMesh_IndexSet([])

Indices in the right set wall set:
FeMesh_IndexSet([ 3,  7, 11, 15, 19, 35])
Available special sets for this mesh are:
dict_keys(['MinI_VertexSet', 'Left_VertexSet', 'MaxI_VertexSet', 'Right_VertexSet', 'MinJ_VertexSet', 'Bottom_VertexSet', 'MaxJ_VertexSet', 'Top_VertexSet', 'AllWalls_VertexSet', 'Empty'])

Indices in the left set wall set:
FeMesh_IndexSet([ 0,  5, 10, 15, 20, 25])

Indices in the right set wall set:
FeMesh_IndexSet([])
Available special sets for this mesh are:
dict_keys(['MinI_VertexSet', 'Left_VertexSet', 'MaxI_VertexSet', 'Right_VertexSet', 'MinJ_VertexSet', 'Bottom_VertexSet', 'MaxJ_VertexSet', 'Top_VertexSet', 'AllWalls_VertexSet', 'Empty'])

Indices in the left set wall set:
FeMesh_IndexSet([])

Indices in the right set wall set:
FeMesh_IndexSet([ 3,  7, 11, 15, 20, 30])