Improve AtomicFileWriter's support for Win and Mac

[n1hility]

As discussed on containers/common#1397, this PR is a step towards addressing corruption issues with container and machine configuration files that can occur with an improper shutdown.

Add ExplicitCommit option to provide calling code the ability to preserve original files when an application error occurs, not just an error that occurs during writing (e.g. a marshaller failure)

Change Close() impl to safely handle redundant calls (for defer usage safety)

Introduces two flush points, one just after data is written that supports the previous code's Linux approach of using fdatasync(), and another for Mac/Win/Non-Linux Unix that needs to full flush after metadata updates are staged.

Using this patch, common's usage for writting the container config file would look like this:

opts := &ioutils.AtomicFileWriterOptions{ExplicitCommit: true}
configFile, err := ioutils.NewAtomicFileWriterWithOpts(path, 0o644, opts)
if err != nil {
	return err
}
defer configFile.Close()

enc := toml.NewEncoder(configFile)
if err := enc.Encode(c); err != nil {
	return err
}

// If no errors commit the changes to the config file
return configFile.Commit()

[vrothberg]

@Luap99 @giuseppe @nalind PTAL

[vrothberg]

I think I'm missing a point and wonder why ExplicitCommit and Commit() are needed. I somehow expected changes to Close() only. @n1hility, can you elaborate on that?

[Luap99]

I think the point is that there is no way to abort this operation.
As shown in the example in the PR description when the encoder returns an error which was not caused by the Write() call, the encoder returns early and we can have a corrupted file once Close() is called. The only way to prevent this would be to not call Close() but this would then leak fds instead and would never allow the use of defer somefile.Close()

[vrothberg]

Thanks, makes sense 👍

[n1hility]

Great summary, I couldn't have said it better. I would have preferred to make a hard move and make this commit() approach the only option instead of a flag like in my PR, since the previous auto-commit behavior is likely to cause problems in most usage patterns. However, it would be compatibility breaking, and I wasnt sure how much usage of this API there is in the wild.

[vrothberg]

I think that's a great way of keeping the API stable, @n1hility. Thanks!

[vrothberg]

LGTM

[rhatdan]

LGTM