LocalEnvironment to stop being IDisposable

[Zruty0]

The only reason we have IHostEnvironment as an IDisposable is because we clean up the temp files created using CreateTempFile. This functionality is not really very useful. We should remove the functionality, remove the IDisposable and fix all the using calls in the codebase (preferably to just new MLContext, if possible)

[TomFinley]

The implication is that anything that relies on it for temporary file creation and that requires must cease to do so. In order for us to do this, we must also remove temporary file creation from IHostEnvironment. (Also, regular file creation, though that is less urgent.) This I have found untangling this relatively easy to do, but there is one exception that is very troublesome because it explicitly relies on this behavior.

machinelearning/src/Microsoft.ML.Data/EntryPoints/Cache.cs

Lines 71 to 73 in 057c4b9

	// We are not disposing the fileHandle because we want it to stay around for the execution of the graph.
	// It will be disposed when the environment is disposed.
	var fileHandle = host.CreateTempFile();

That's bad, but I think there's a pretty easily solution. I think it is completely reasonable for someone asking for disk cache to provide a place where on the disk it should be stored. I think that will actually be an improvement -- certainly it is not an expectation that any "temporary" files will be many gigabytes or terabytes large (which is the only conceivable reason why anyone would use disk caching in the first place), so having some ability to configure where it is stored would I think be good. Or we provide access to the binary saver via entry-points and say that if someone wants that functionality they just have to do it themselves. This I think is fine, since entry-points were entry-points into ML.NET for wrapping code, not user facing code.

[glebuk]

@TomFinley is this still relevant? IHostEnvironment is no longer disposable AFAIK.

[eerhardt]

Yes, this is still relevant.

We still have these methods on IHostEnvironment that we need to remove:

machinelearning/src/Microsoft.ML.Core/Data/IHostEnvironment.cs

Lines 55 to 77 in 0c62e30

	/// <summary>
	/// Return a file handle for an input "file".
	/// </summary>
	IFileHandle OpenInputFile(string path);
	/// <summary>
	/// Create an output "file" and return a handle to it.
	/// </summary>
	IFileHandle CreateOutputFile(string path);
	/// <summary>
	/// Create a temporary "file" and return a handle to it. Generally temp files are expected to be
	/// written to exactly once, and then can be read multiple times.
	/// Note that IFileHandle derives from IDisposable. Clients may dispose the IFileHandle when it is
	/// no longer needed, but they are not required to. The host environment should track all temp file
	/// handles and ensure that they are disposed properly when the environment is "shut down".
	///
	/// The suffix and prefix are optional. A common use for suffix is to specify an extension, eg, ".txt".
	/// The use of suffix and prefix, including whether they have any affect, is up to the host environment.
	/// </summary>
	[Obsolete("The host environment is not disposable, so it is inappropriate to use this method. " +
	"Please handle your own temporary files within the component yourself, including their proper disposal and deletion.")]
	IFileHandle CreateTempFile(string suffix = null, string prefix = null);

Even though it doesn't implement IDisposable, we can't have these methods on this interface.

[eerhardt]

Just an FYI - it looks like the only place in the product that is calling Cache.CacheData (which is the only place calling CreateTempFile pointed out by @TomFinley above) is coming from Entry-Points:

machinelearning/src/Microsoft.ML.Data/EntryPoints/InputBase.cs

Lines 220 to 224 in 174de1a

	var cacheView = Cache.CacheData(host, new Cache.CacheInput()
	{
	Data = roleMappedData.Data,
	Caching = cachingType.Value
	}).OutputData;

This is good, because it means you can't get to this code from public APIs. And we can force the EntryPoints to call Dispose to clean up the temp files.