-
Notifications
You must be signed in to change notification settings - Fork 39.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Capture application termination messages/output #139
Comments
Is /run a tmpfs? There is an outstanding PR for this in libcontainer. |
Good call. Looks like not. Here's df from a google/nodejs container: |
Is /run LSB compliant? |
It will be once LSB is updated to FHS 3.0: http://askubuntu.com/questions/57297/why-has-var-run-been-migrated-to-run |
Re. /run: The point of tmpfs was to avoid pathological disk latency and failure problems. However, we'd need the filesystem to remain live after termination of the main process. We want that for other reasons (e.g., hooks), but it doesn't exist yet. Solomon expressed some interest in this on #docker-dev: |
One could also view this as simple container output. I could imagine using this for simple data-in/data-out functions, such as config generators. One question would be whether we should make the path configurable and, if so, should we provide a means to tell the container what that path is? I could imagine allowing the user/client to specify the path and environment variable name. However, I could also envision standardizing it for containers, potentially even beyond just Docker containers. For instance, could we use /dev/console, similar to VM console output in GCE? Or maybe another file in /dev. Note also that /dev/stdout is linked to /dev/fd/1, /dev/stderr is linked to /dev/fd/2, and /dev/ptmx is linked to /dev/pts/ptmx. |
/dev/console is used by some images/distributions, so it probably needs to be /dev/somethingelse. |
Possible file names: /dev/stopmsg, /dev/finalstatus, /dev/deathrattle, ... /cc @rjnagal |
/dev/log is used by syslog. How about /dev/final-log? |
@bgrant0607 I assume we want to have a structured logging format for the death reason. If we were to define and provide a new interface, how can we promote adoption of this interface? Requiring application changes might hinder adoption. |
No, I don't want a structured logging format. I want the raw output. We can capture other termination information (e.g., time and reason) separately. With respect to usage: We should ensure that it is easy for a user to add a PreStop hook to populate it. As for automatic extraction from Docker logs: I'd want to strip the cruft and display the raw output. But how many lines? Fatal log messages are typically 1 line but stacktraces and uncaught language exceptions may be many lines. Rather than building this functionality into Kubernetes, we could provide a script or program that the user can mount into their container and run as a hook, with a configurable number of lines. In terms of promotion: I'd like to see Docker, libcontainer, and the container community more broadly adopt a mechanism like this. The "container RFC" should have proposed something like this. |
A PostStop hook would probably work better than PreStop. |
Structured logging might provide ability to make restart decisions which Docker logs: I get that it is difficult to ascertain the number of log If the applications were to dump their death reason to a location that is On Tue, Nov 4, 2014 at 3:29 PM, bgrant0607 notifications@github.com wrote:
|
@bgrant0607 how about /dev/termination_log? Put some thoughts on the issue this afternoon, and here is the rough design / proposal:
|
@vishh @dchen1107 and I discussed this in person. First of all, structure: Different kinds of information need to be communicated:
We can definitely leave affordances in the API (in We're going to punt on at least (4) and probably (3) for now. I feel (2) is most important, but (1) is also widely useful. Since I believe there is no clear line between infrastructure and user control components, I don't feel we should differentiate Kubelet-originated (e.g., SystemOOM) and other reasons (e.g., WatchDogTimeout) by using separate fields. We should also generate events for eventualities the system should respond to, such as system OOM. On the number of lines to pull from logs by default: Docker is at least planning to move to a write-oriented log model rather than a line-oriented model, which may solve this problem. Regarding the file location: Using a /dev location would mean that we'd need to manage the bind mounts in the user containers. Using a standard location would require application changes or adapter hooks. Regarding the specific name, termination_log is a bit long, but I agree it's more consistent with the other terminology. I'd use a hyphen rather than an underscore (/dev/termination-log), however. Using a configurable path would require that we check whether it's in a volume or in the container's writable layer until Docker decouples the mount namespace lifetime from the main process lifetime. We'd copy from the host filesystem for the former and docker cp for the latter. Setting the configurable path to, say, a glog Fatal log location would provide the data for (2) only. We'd have to synthesize the reason, which could, for instance, be "FatalLog". However, I don't see a good way to accurately characterize the failure reason without running some characterization code. We could provide a default characterization program for common formats, such as glog, Java exceptions, etc. Finally, we might as well capture the termination-log even for successful termination. The application might emit a brief execution summary of some sort. Full-blown application output should be handled via a different mechanism. |
@bgrant0607 what you commented above is aligned with my initial proposal. The only change I made is taking @vishh's suggestion to allow the user configured a path. |
…eate Revert "grpc for CreatePod"
Update links and info for sig-api-machinery
Scale the deployment webserver to 6 pods kubectl scale deployment/webserver --replicas=6 |
Fix deployment of recent bash
* add hello-app-redis * Update app-deployment.yaml * Update main.go * resolve comments
…-controllers Compile fixes
update README
…atches [release v1.28] k8s v1.28.11
When applications terminate, they may write out important information about the reason, such as assertion failure messages, uncaught exception messages, stack traces, etc. We should establish an interface for capturing such information in a first-class way for termination reporting, in addition to whatever is logged.
I suggest we pull the deathrattle message from /dev/final-log or something similar.
The text was updated successfully, but these errors were encountered: