Configure Elastic Agent host path volume to point to correct path #5890
Conversation
There was a problem hiding this comment.
Since we are considering to flag this as a breaking change, and since Agent is still considered as "experimental", I'm wondering if we should not change the path on the host node's filesystem
- hostPath:
path: /var/lib/<ns>/<name>/agent-data
type: DirectoryOrCreateI never realised that the first directory created by the operator is named according to the namespace name. However /var/lib is not exclusively used by ECK, I'm then wondering if we should not consider the case where there might be a conflict with an existing directory. For example on GKE nodes there is an existing directory named /var/lib/metrics. Creating an agent in a namespace named metrics would store Agent data in a directory that might have different purposes and lifecycle. Also it does not seem to be consistent with the naming from the Agent project which is /usr/share/elastic-agent/...
Should we change the directory path on the host to something like /usr/share/elastic-agent/server/<ns>/<name>/state ?
Yes I think that sounds reasonable.
Why the server bit? |
|
Also Should we do |
I was wondering if it would make sense to add the Fleet "type" in the path, but you're right it's not required.
The intent was to be consistent with the path in the Pod. It is however definitely not a strong opinion and you're right, I think |
Maybe we could mention it in our documentation as a "limitation"? |
|
run/e2e-tests tags=agent |
|
run/e2e-tests tags=agent |
david-kow
changed the title
Mount the host path volume to point to the path where Elastic Agent stores its local state. This should make sure that while running on the same version no duplicate ingestion is happening. After version upgrades the internal sub-paths within the state directory still change see elastic/elastic-agent#750 and events will be re-ingested. This is a potentially breaking change as I am configuring the host path volume now independently of the mode the agents are running in: Fleet or stand-alone. It will be breaking for users who have configured a deployment with multiple replicas where more than one replicas is placed by the Kubernetes scheduler on the same K8s node.
Fixes #4428
Mount the host path volume to point to the path where Elastic Agent stores its local state. This should make sure that while running on the same version no duplicate ingestion is happening. After version upgrades the internal sub-paths within the state directory still change see elastic/elastic-agent#750 and events will be re-ingested.
This is a potentially breaking change as I am configuring the host path volume now independently of the mode the agents are running in: Fleet or stand-alone. It will be breaking for users who have configured a deployment with multiple replicas where more than one replicas is placed by the Kubernetes scheduler on the same K8s node.
Can we mitigate the effects of this? We could optimistically assume that Deployments with multiple replicas will be mainly used for Fleet Server and not use the host path volumes in this case.
I tried to think of a test case but it is tricky without restarting Elastic Agent Pods to see if duplicates are ingested or not and no clear indication of ingest state (i.e. have we waited long enough for duplicates to appear etc)