Add support for async / await

[roryap]

I'm just wondering why there isn't support for async/await, e.g. in Lucene.Net.Search.IndexSearcher, and if adding that is a goal of the project.

[jeme]

I think it is safe to say that it is not within the scope of this project at the moment. (and probably never will be) Being a Java port it's hard to imagine that such a thing could ever be maintained, that would require a massive effort.

The thing is, if you don't bring it all the way through to the layers where your waiting for outside "interrupts" e.g. when you perform IO or waits for remote services, then it won't really bring any advantages that you can't add your self just by making a proxy around the classes you wish to have working with async/await. (And it's hard to argue that there would be any benefit at all)

I would be all for it, but we have to be realistic.

[roryap]

I get it. It is a shame though. To be scalable in a web server sense, using threads for each search operation where they can be efficiently shared awaiting I/O or other interrupts is crucial.

[rclabo]

I agree that adding async/await doesn't seem feasible. But given how fast lucene is I suspect it can support much more scalability on the web server then one might guess. It'd be cool is someone in the community pushed it on this dimension and reported back their findings.

[eladmarg]

the current Lucene version isn't write in a modern high performant app manner.
yes, its getting stable, its working, but its far from being optimal.

the whole design should be rewritten from bottom to up, many other optimizations are missing.
I'm not sure its even possible in the current code base, because all the data should be async/await all the way from the ground.

this is a big change that will take Lucene to another direction from the port and I'm not sure its in this frame.

however, its nice to have this kind of requests, we all want it but its not possible in this project constraints.

[rclabo]

If one really wanted to influence the direction of Lucene in an effort to get async/await support all the way to the metal, then that should happen in the Java Lucene version first. Then eventually it will get ported to Lucene.NET.

[NightOwl888]

It is difficult to imagine both maintaining a sweeping architecture change as well as keeping pace with Lucene's release schedule. It would presume that one of us has an advanced high-level understanding of the architecture to be able to make and then maintain such a change, but given that this is a low-level line-by-line port, it is not an understanding that anyone on the team has. Our focus is on keeping the project upgradable in the future to integrate new changes to Lucene, which unfortunately means that if we drift too far from the original source we would need a very detailed roadmap to follow to make each upgrade possible.

I have to agree with @rclabo, async/await is a pattern supported in Java, so the best way to support it in .NET would be to lobby the Lucene team to make the design changes at a high level first, which would be propagated to Lucene.NET when it is upgraded. In fact, I would make that argument about any significant high-level design change that isn't specifically made to address a gap between the platforms. And do it immediately, since Lucene is in the process of planning for version 9.0 right now.

The only tool we have that helps us solve some of the failing tests is to run the code both in Java and in .NET to see where the paths diverge between the two. Once we lose the Java reference to work from, some bugs that are easy to identify now would effectively mean researching several high-level design concepts just to understand what the bug is before determining where it is, since we would be unable to distinguish between a bug that is due do to a difference between the Lucene behavior and the Lucene.NET behavior from a bug that was introduced in the new design.

[rclabo]

I agree with @NightOwl888 wholeheartedly. It's nearly impossible to fully express how important it is to have Lucene.NET stick as close as possible to Java Lucene. There is very little code on this planet that has been as battle hardened as the code in Java Lucene. When you consider that it’s the foundation that both Solr and ElasticSearch are built on and then think about the size of the installed base of those products, it’s mindboggling how much of the planet runs and depends on Java Lucene. That level of attention and use over two decades leads to an amazingly solid code base. A hard won result that is not something to be taken lightly or easily achieved by others.

The fact that Java Lucene’s code is so battle hardened is an enormous asset when you stop and consider the size of the code base and factor in that it supports multi threaded concurrent use. Not to mention the fact that many of the concepts implemented in Lucene are indeed state of the art and non-trivial to understand. Taken together a senior developer would need to work with this codebase for 5 – 10 years to come anywhere close to fully comprehending it. And without that level of knowledge, any significant changes or deviations from Java Lucene are likely to result in bugs that could be very difficult to track down.

I believe that it's absolutely vital that Lucene.NET stay true to its goal of being a line by line port other than minor API changes to adhere to .Net naming conventions and such where applicable. Even now there are parts of Lucene.NET that I wish tracked closer to the Java code. In some cases that’s just not possible because of the differences in the platform tools but our goal non the less should be to port Lucene.NET from Java Lucene as faithfully as possible.

[roryap]

@NightOwl888 @rclabo perfect explanations. Indeed, if Java Lucene were to take on async/await, it would have to be done in a way that doesn't affect the existing code base or else risks breaking it. I can only wonder at how much work it would be to erect a parallel code base that uses async/await from the ground up.

[rclabo]

@roryap It's really hard to say, but such a code base would likely result in a lot of bugs that take a very long time to work out.

Before one should walk down that path they should verify carefully that lack of async/await is actually the source of throughput bottleneck in their desired use case. Are they really saturating the machine with threads and truly running out of the ability to create new ones while the majority of threads are doing nothing but waiting on disk IO? Verifying that such a situation truly exists should be job one before even thinking of making a parallel modified codebase that one must maintain by themselves. It's certainly not a road I'd want to walk down.

[rclabo]

@roryap As a side note, we are always looking for other developers help with Lucene.NET. While the current 4.8 beta is very stable more work is needed to fix bugs, resolve issues, improve docs and polish it for release. We'd welcome your participation if you are interested.

[NightOwl888]

I am closing this for now, since it is definitely an issue for the Lucene team to tackle, not for us.

@roryap

Do note that work is underway right now with the Lucene 9.0 release, and 9.x is likely the next point we will be syncing with them. If you want to see this happen in a future release of Lucene.NET, I urge you to start a discussion with the Lucene team if you haven't done so already.