What is color relativity and how do I start using it in my own art?
Color relativity concerns how one color appears compared to another. It derives from the observation that the appearance of a color in a scene is actually dependent on the colors surrounding it. 
For example, consider a green bowl surrounded by oranges. What color should you pick for the bowl's shading? You might be tempted to think that the darker parts of the green bowl will be a darker green. However, compared to the oranges surrounding the green bowl, that dark shadow may actually appear more blue, and you may achieve a more natural looking image by picking a shade that leans towards blue. 

When working on your own art, see if you can find references that depict the kind of lighting you are looking to create. Try to break your reference down into large blocks of common color (e.g. the shaded parts of our green bowl may be one block of color, while the lit part may be another); this should make it easier to see how the colors appear, as opposed to what you know them technically to be.
How does it work actually? Do the color relations have some order? What kind of mathematical structure comes closest to the color relativity in color space?
Color relativity is an optical illusion caused by imperfections in our eyes. Since perception can vary from human to human, there is no solid mathematic function that can model the human perception of color. Generally though, humans differentiate colors based on value (black vs white), then saturation (monochrome vs color), then by hue (red vs blue vs green vs yellow)