Can we allow FSDirectory subclasses to customize whether the ctor does a mkdir? [LUCENE-7375]

[asfimport]

Today, just instantiating an FSDirectory always brings the directory into existence, even if you will do no writes (createOutput).

We have gone back and forth on this, over the ages. E.g. see #1094 (only 2 digits there!, #1848 (only 3 digits there)!), #2538. At one point we created the directory lazily, on the first write (createOutput) attempt, but now we always create it when you instantiate FSDirectory.

This causes some hassle for consumers, e.g. in elastic/elasticsearch#19338 ES is forking SimpleFSDirectory in order to have a read-only directory impl.

Maybe we can do the Files.createDirectories in protected method that a subclass could override?

---

Migrated from LUCENE-7375 by Michael McCandless (@mikemccand)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

the mkdir is necessary if the Directory will ever be used to perform any writes, otherwise locking is broken.

Lucene was broken here forever, because it used getCanonicalPath on a directory that may or may not exist: that would then be used by e.g. NativeFSLock as the key for the in-memory map and so on.

This was just one of the many ways that locking was broken.

You can see this by reading the javadocs:
https://docs.oracle.com/javase/7/docs/api/java/io/File.html#getCanonicalPath()

Every pathname that denotes an existing file or directory has a unique canonical form. Every pathname that denotes a nonexistent file or directory also has a unique canonical form. The canonical form of the pathname of a nonexistent file or directory may be different from the canonical form of the same pathname after the file or directory is created. Similarly, the canonical form of the pathname of an existing file or directory may be different from the canonical form of the same pathname after the file or directory is deleted.

So, in order for locking to work, its critical that we create the file.

[asfimport]

Uwe Schindler (@uschindler) (migrated from JIRA)

Maybe allow to pass "read-only" to FSDirectory ctor and FSDirectory.open()? In the RO case we can fail early if the directory does not exist. In addition, all the write methods fail with UOE.

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Well, another option is that NativeFSLockFactory etc already do their own createDirectories + toRealPath (defensively). This came after the move to nio.2, and maybe the one in FSDir can now just be removed.

This would require a bit of hair though: I think the only clean way to do it, would be to say that the lockfactory is responsible for creating the directory if it doesnt exist. That has implications for NO_LOCK_FACTORY and other crazy options we have :)

[asfimport]

Robert Muir (@rmuir) (migrated from JIRA)

Also, think about any external use of getDirectory() and so on (this is even used by toString). If someone calls this method, e.g. to key into a map of their own, we've brought the old java.io.File trap back :)

So the Files.createDirectories isn't useless in that sense either, it returns toRealPath today, which means such traps are impossible.