how to configure image annotation that use RGB code as class identifier

[ramdhan1989]

my image annotation is in RGB color. notebook example in this repo is using label encoding image (CamVid dataset). I got problem when trying to convert to label encoding image so that I can continue to the next code hopefully without any further code modification.
this is the example of label annotation :
https://drive.google.com/file/d/1h1Ezc1QobHVlttF3P53yH2V8e5bMq8yL/view?usp=sharing

this is the code
CLASSES = ['pipeline','person','road','house','car','cargo truck','oil truck','motorcycle'] #all clasess
label_values = [[255, 0, 0],[0, 0, 255],[255, 255, 0],[0, 255, 255],[255, 0, 255],[128, 0, 128],[128, 128, 0],[0, 0, 128]]
classes = ['pipeline','person','road','house','car'] #select only certain classes
class_values = [CLASSES.index(cls.lower()) for cls in classes]
mask = cv2.cvtColor(cv2.imread('.../example_annot.png'), cv2.COLOR_BGR2RGB)
semantic_map = []
for colour in [label_values[index] for index in class_values ]:
equality = np.equal(mask, colour)
class_map = np.all(equality, axis = -1)
semantic_map.append(class_map)
mask2 = np.float32(np.stack(semantic_map, axis=-1))
mask3 = np.argmax(mask2, axis = -1)
masks = [(mask3 == v) for v in class_values]
mask4 = np.stack(masks, axis=-1).astype('float')

mask is shown below

let's focus on pipeline class (red color in original image) as shown by mask2 (plt.imshow(mask2[:,:,0])) below

now check label encoding image result as shown by mask3

the pipeline object is gone !
recheck from mask4 for pipeline class

mask4[:,:,0] now become background rather than pipeline class

I don't understand what heppened here. is there anyone can explain it ?
I would like to have advice also how to deal with image annot that use RGB code as class identifier that easly to adapt with whole script provided in this repo ?

thank you