Add service metadata monitoring

[samjsong]

Decided this was a really stupidly large PR (mostly UT's and JSON I promise). Really the only big addition is just service_monitor.rb.

I used much of the watch thread logic from the Events plugin, but we don't need to persist state on restart since we don't care about data duplication.

Next step is to actually read from this hash and attach as metadata to the record. Decided to put that on hold until we make the changes in SUMO-112136 which will make this part easier.

If anything doesn't make sense or it looks inefficient or messy please let me know and I will fix 🙂 For an idea of what input and output looks like, feel free to look at the test JSONs and UT's.

[samjsong]

Hmm build is broken even though ci/build.sh was running fine on my local computer before I merged master into the branch, but now I'm able to reproduce locally. Looking into it

[samjsong]

Okay the issue is that in test_docker.rb it's trying to create a kubeclient and start a watch call to the API except there's no kubernetes url set since it's just a dummy image. If you set the env vars for a dummy kubernetes url when starting the docker image in test_docker.rb, then it gets stuck trying to create the client.
(If you change

<filter FOR_TEST_ONLY>
  @type enhance_k8s_metadata
</filter>

to

<filter FOR_TEST_ONLY>
  @type events
</filter>

it also fails, since the events plugin also tries to start a watch thread. I'm going to remove the dummy test for now, we can consider how to fix this later.

[maimaisie]

not sure if I'm following the MODIFIED case, can you give some explanation?

[samjsong]

Yeah, the MODIFIED case was the hardest part. There's two cases to cover:

1. The service was modified such that there are more pods than there once were (increasing the number of replicas). This is the easier case. We simply add the service to @pods_to_services[pod] unless it already exists.
2. The service was modified such that there are now fewer pods than there once were (downscaling the number of replicas). This is the harder case, since you don't know which pods were removed you have to actually search the map for mentions of the service, and delete it if it's not one of the pods we were told about in the MODIFIED event. Then if there are no services for that pod in the map, we can remove that key from the map.

I offhandedly asked Chris about it at some point, since the "looking through the entire map for mentions of the service" part really irked me... but he seemed to think it wasn't that big of an issue to be worth "over-optimizing prematurely", since this case likely will not happen often. I'm open to any suggestions 😅

[maimaisie]

thanks for the detailed explanation! I'm fine with current approach

[maimaisie]

does deleting service from services equivalent to deleting it from @pods_to_services[pod]?

[samjsong]

Yes, I believe services is a reference to the array in the value