++, I suggest this setting will be real time setting (changed using index update settings API). I think the interesting question would be into which file will it be logged? I think it should be a separate file, that is placed in the same logging dir location.

I like the idea of having this be a real-time setting, so you can go into an existing index that's doing something strange and turn on IW.infoStream. The challenge is, this is not a real-time setting in IndexWriter today (you must close/reopen the writer), which is annoying.

I think we should just make our own delegating InfoStream impl that we pass to IW on init, so IW always logs messages to it, but then we tell it to turn on/off whether it sends those messages to a log file (I agree, it should be a separate file in the normal logging dir).

I'll give this a shot.

Wondering if a delegate InfoStream will not hurt perf, since the string construction to log it will happen now all the time within Lucene. Potentially, this setting can be realtime, but we will close and open the IndexWriter with it within our Engine (we can do that with some settings that require a fresh IW).

This can safely be a realtime setting.

There is actually always a infoStream set, it is just a NullInfoStream that just always returns false for isEnabled? All checks in IndexWriter look like this:

  if (infoStream.isEnabled("DWPT")) {
    infoStream.message("DWPT", "done abort");
  }

So its sorta like logger apis in that sense. You can safely have one that flips on and off at will:

About linking InfoStream to a logger, it should be pretty simple. We added this to solr with this class: http://svn.apache.org/repos/asf/lucene/dev/trunk/solr/core/src/java/org/apache/solr/update/LoggingInfoStream.java

However, there was more going on there (e.g. legacy settings for infostream and so on).

Maybe for ES, there should be no option for infostream at all? it should just be set, and it should check if trace or debug level logging is enabled?

Basically: the user would be unaware of infostream. They just crank up logging and see more stuff...

I like that solution; this way the user doesn't need to learn another setting. This is just increased verbosity.

we could have it under a specific logging name, yea, and then its just a matter of enabling trace logging on it specifically. I missed the enhancement to InfoStream, nice!

OK, I pushed an initial commit .... I found a LoggerInfoStream which
didn't seem to be used and whose intent seemed to be the same as this
issue (just for an older time when Lucene took PrintStream), so I
re-purposed it into a Lucene InfoStream.

I'm not sure about passing the indexSettings/shardId down, but I
want to make sure the shardId is in the log so we can separate out
the logs from multiple IndexWriters. Is there a cleaner way?

I also hit & "fixed" an unrelated NPE when I set logging to trace but
I'm unsure if it's a problem that newSearcher is null when topReader
was true? I put a nocommit.

I turned on trace logging in the tests (-Des.logger.level=TRACE) and
confirmed the IndexWriter's infoStream messages are appearing. I
still need to figure out how to make a test case that checks
it... maybe I need to set up a MockAppender...