docs: inotify behavior on atomic configMap/secret volumes is not documented

[ahmetb]

Background:
Kubernetes automatically updates files on mounted secret and configMap volume types, as respective Secret/ConfigMap resources change.

Problem:
When someone runs a program on Kubernetes watching filesystem updates on secret and configMap volume types volumes (which is typically done using inotify(7) syscall on POSIX systems), they will be getting only "file deleted" (IN_DELETE_SELF) inotify event when the files are updated. (This is because how Kubernetes/kubelet uses AtomicWriter to make updates to these files.)

This behavior is quite different than what the program would see when it's running outside Kubernetes, since a normal filesystem would return inotify events like IN_CLOSE_WRITE (file opened for writing closed) or IN_MODIFY (file written) events. Furthermore, "file deleted" event causes the inotify monitor to break. Therefore, the code needs to handle this to re-establish the inotify watch upon deletion (otherwise it would only receive the one delete event, and nothing else).

This caveat is not documented.

Solution:
Since this behavior might break existing programs that watch files when the program is migrated to run on Kubernetes, it would be ideal to note this pitfall in the project documentation (this page?).

It currently seems that this information is tucked away in some blog posts [1], [2], [3] and would be good to surface in official documentation as well.

/kind bug
/area docs

[k8s-ci-robot]

@ahmetb: The label(s) area/docs cannot be applied, because the repository doesn't have them.

Details

In response to this:

Background:
Kubernetes automatically updates files on mounted secret and configMap volume types, as respective Secret/ConfigMap resources change.

Problem:
When someone runs a program on Kubernetes watching filesystem updates on secret and configMap volume types volumes (which is typically done using inotify(7) syscall on POSIX systems), they will be getting only "file deleted" (IN_DELETE_SELF) inotify event when the files are updated. (This is because how Kubernetes/kubelet uses AtomicWriter to make updates to these files.)

This behavior is quite different than what the program would see when it's running outside Kubernetes, since a normal filesystem would return inotify events like IN_CLOSE_WRITE (file opened for writing closed) or IN_MODIFY (file written) events. Furthermore, "file deleted" event causes the inotify monitor to break. Therefore, the code needs to handle this to re-establish the inotify watch upon deletion (otherwise it would only receive the one delete event, and nothing else).

Solution:
Since this behavior might break existing programs that watch files when the program is migrated to run on Kubernetes, it would be ideal to note this pitfall in the project documentation (this page?).

/kind bug
/area docs

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@ahmetb: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ahmetb]

/remove-label bug
/sig docs
/kind documentation

[k8s-ci-robot]

@ahmetb: The label(s) /remove-label bug cannot be applied. These labels are supported: api-review, tide/merge-method-merge, tide/merge-method-rebase, tide/merge-method-squash, team/katacoda, refactor, official-cve-feed

Details

In response to this:

/remove-label bug
/sig docs
/kind documentation

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[BenTheElder]

Nits:

since a normal filesystem would return inotify events

It's a "normal file system", the behavior is entirely in file manipulation in ways that can happen on any filesystem. (I get this is mostly semantics, but that matters for the docs!)

Code that only triggers on IN_CLOSE_WRITE can be broken by this technique outside of Kubernetes. using rename(2) for atomic file changes is a documented use case:

One useful feature of rename is that the meaning of newname changes “atomically” from any previously existing file by that name to its new meaning (i.e., the file that was called oldname). There is no instant at which newname is non-existent “in between” the old meaning and the new meaning. If there is a system crash during the operation, it is possible for both names to still exist; but newname will always be intact if it exists at all.

https://www.gnu.org/software/libc/manual/html_node/Renaming-Files.html#Renaming-Files

https://lwn.net/Articles/789600/ has a discussion of even adding new ways for applications to do this.

I do think this is something we should mention that simple inotify probably won't work, but I'm not sure we should document the implementation details vs suggesting reliable alternatives that don't depend on the implementation (like periodically reading and diffing vs the last read).

I also wouldn't be surprised to see other systems leveraging this known-technique outside kubernetes for the same reason Kubernetes does ... to prevent the even worse pitfall of clients reading partially-rewritten content.

[BenTheElder]

/sig node storage

[ahmetb]

The main problem is the class of programs reloading configuration based on the filesystem writes that are probably handling a "delete" event by printing an error saying "configuration file no longer exists, exiting" and subsequently issuing an exit() redundantly (whereas they could just attempt to retry the watch).
Maybe I am exaggerating the problem irl, but a program that's purely looking at write/update events won't get any of that on Kubernetes –and that's notable?