Handle SD_JOURNAL_INVALIDATE when following a journal

[nalind]

- What I did
This is taking a stab at fixing the open-file-handles-for-deleted-journal-files parts of #27343.

- How I did it
We weren't handling SD_JOURNAL_INVALIDATE results coming back from sd_journal_process() before, so I added logic to close and reopen the journal handle when we get that result.

- How to verify it
Run docker logs -f to tail the journaled output from a container that was started with --log-driver journald, and consult the list of open descriptors in the directory under /proc/PID/fd which corresponds to the docker process to see which journal files it has open.
While docker logs -f is running, use journalctl --vacuum-size or journalctl --vacuum-files to force the removal of one or more older journal files which dockerd had open, then consult its fd directory again: it should no longer have open descriptors for files which were removed (i.e., the kernel should not be noting any journal files which it has open as (deleted)).

- Description for the changelog

When following a journal in the journald log reader, correctly handle SD_JOURNAL_INVALIDATE results from sd_journal_process() by closing and reopening the journal handle, which closes all open journal files (including deleted ones) and opening all currently-available journal files.

[cpuguy83]

Looking through the code, it seems this will terminate the follow regardless of the container exit or user request in this case (which it already did anyway, so nothing new). This doesn't seem quite like the correct behavior.

[nalind]

@cpuguy83 I think that's the behavior we get with json-file as well.

[cpuguy83]

The json logger gets rotation events and re-opens the log.

[nalind]

@cpuguy83 Right. I think I'm misunderstanding what you're saying here - the reader should be continuing to follow the journal after it handles the SD_JOURNAL_INVALIDATE result.

[cpuguy83]

Oh you're right, I guess I need to step away for a bit and come back to it.

[cpuguy83]

LGTM

[boaz0]

LGTM 👍

[portante]

FWIW, this is an example C program for what it takes to tail the journal without duplicates and not leaking file handles on journald rotation: https://gist.github.com/portante/ff7fb429c6f973aab377f7bb77b0ffdb

[nalind]

Hmm, further testing is turning up that we need to re-seek to the point where we were reading when we reopen the journal, and I'm seeing some blocking when writing messages to the logWatcher.Msg channel if a following client disconnects while we're doing a long read, so I think this needs some more work.

[nalind]

I'm going to open a fresh PR for an updated patch, since reworking it to make more frequent calls to sd_journal_process() ends up undoing what we were proposing to change here.

[nalind]

Opened #36254 to try to solve this properly.