Web app and Rest API: Support for multiple indexes #2486

16BitNarwhal · 2024-05-05T16:15:33Z

New api route /api/collection/{collection}/search?query={query}

Web app allows selecting different prebuilt-indexes from dropdowns

codecov · 2024-05-05T16:25:06Z

Codecov Report

Attention: Patch coverage is 22.22222% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 66.63%. Comparing base (2331882) to head (80da988).
Report is 7 commits behind head on master.

❗ Current head 80da988 differs from pull request most recent head 6b475b6. Consider uploading reports for the commit 6b475b6 to get more accurate results

Files	Patch %	Lines
...rc/main/java/io/anserini/server/SearchService.java	0.00%	5 Missing and 1 partial ⚠️
src/main/java/io/anserini/server/Controller.java	66.66%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2486      +/-   ##
============================================
+ Coverage     66.43%   66.63%   +0.19%     
- Complexity     1424     1432       +8     
============================================
  Files           215      215              
  Lines         12367    12378      +11     
  Branches       1504     1511       +7     
============================================
+ Hits           8216     8248      +32     
+ Misses         3632     3610      -22     
- Partials        519      520       +1

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lintool · 2024-05-05T17:25:07Z

Hi @16BitNarwhal Running this:

Getting following exception:

2024-05-05 13:23:47,529 DEBUG [http-nio-8081-exec-1] log.LogFormatUtils (LogFormatUtils.java:120) - GET "/api/collection/msmarco-v1-passage.splade-pp-ed/search?query=Albert%20Einstein", parameters={masked}
2024-05-05 13:23:47,530 DEBUG [http-nio-8081-exec-1] handler.AbstractHandlerMapping (AbstractHandlerMapping.java:531) - Mapped to io.anserini.server.Controller#search(String, String)
Index file already exists! Skip downloading.
Index folder already exists!
May 05, 2024 1:23:47 PM org.apache.lucene.store.MemorySegmentIndexInputProvider <init>
INFO: Using MemorySegmentIndexInput with Java 21; to disable start with -Dorg.apache.lucene.store.MMapDirectory.enableMemorySegments=false
java.lang.IllegalArgumentException: argument "content" is null
	at com.fasterxml.jackson.databind.ObjectMapper._assertNotNull(ObjectMapper.java:5054)
	at com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:3276)
	at io.anserini.server.SearchService.lambda$search$0(SearchService.java:45)
	at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:197)
	at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
	at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:509)
	at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:499)
	at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:921)
	at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:682)
	at io.anserini.server.SearchService.search(SearchService.java:58)
	at io.anserini.server.Controller.search(Controller.java:27)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:255)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:188)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:118)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:925)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:830)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1089)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:979)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1014)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:903)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:564)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:885)
	at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:658)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:205)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:116)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:174)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:149)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:167)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:482)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:115)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:344)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:391)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:896)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1744)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)

Any ideas?

16BitNarwhal · 2024-05-06T02:12:49Z

The cause seems to be that the documents in SimpleSearcher don't always contain the content for some of the indexes like msmarco-v1-passage.splade-pp-ed, a sample document stored

as opposed to msmarco-v1-passage which gives

I'm getting the documents via searcher.doc(result.docid) or searcher.doc(result.lucene_docid) given that searcher is a SimpleSearcher and result is a ScoredDoc from searcher.search()

lintool · 2024-05-06T02:14:24Z

Ah, good point. I remember now. Let me mull a bit on how to best address this issue...

lintool · 2024-05-12T03:03:20Z

src/test/java/io/anserini/server/ControllerTest.java

@@ -0,0 +1,20 @@
+package io.anserini.server;


boilerplate

lintool · 2024-05-12T03:03:29Z

src/test/java/io/anserini/server/ControllerTest.java

+
+public class ControllerTest {
+
+    @Test


two space indent please

lintool · 2024-05-12T03:05:34Z

src/main/java/io/anserini/server/Controller.java

+
+  @RequestMapping(method = RequestMethod.GET, path = {"/collection/{collection}/search", "/search"})
+  public List<QueryResult> search(
+      @PathVariable(value = "collection", required = false) String collection, 


I use a line width of 120 characters, so can go on previous line?
I care more about "looking right" than observing strict limits.

16BitNarwhal and others added 6 commits April 30, 2024 12:13

modify api route

841bf25

webapp for multiple indexes

8e2b866

added all indexes and grouped them

84319bd

multilevel dropdown and changed grouping

2313c32

dropdown update

1803547

Merge branch 'castorini:master' into web-app

6e67db8

16BitNarwhal marked this pull request as ready for review May 5, 2024 16:25

16BitNarwhal added 2 commits May 11, 2024 13:32

add server ControllerTest

a600174

dropdown update

80da988

lintool reviewed May 12, 2024

View reviewed changes

src/test/java/io/anserini/server/ControllerTest.java

@@ -0,0 +1,20 @@

package io.anserini.server;

Copy link

Member

lintool May 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

boilerplate

lintool reviewed May 12, 2024

View reviewed changes

src/test/java/io/anserini/server/ControllerTest.java Outdated

public class ControllerTest {

@Test

Copy link

Member

lintool May 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

two space indent please

lintool reviewed May 12, 2024

View reviewed changes

16BitNarwhal added 3 commits May 12, 2024 12:02

api test index not found

4c56985

fixes

7448b65

fix

0d827b0

lintool approved these changes May 12, 2024

View reviewed changes

add boilerplate

6b475b6

16BitNarwhal merged commit 267f05a into castorini:master May 13, 2024
1 check passed

16BitNarwhal deleted the web-app branch May 13, 2024 00:21

lintool mentioned this pull request May 14, 2024

REST API design #2481

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Web app and Rest API: Support for multiple indexes #2486

Web app and Rest API: Support for multiple indexes #2486

16BitNarwhal commented May 5, 2024

codecov bot commented May 5, 2024 •

edited

Loading

lintool commented May 5, 2024

16BitNarwhal commented May 6, 2024

lintool commented May 6, 2024

lintool May 12, 2024

lintool May 12, 2024

lintool May 12, 2024

Web app and Rest API: Support for multiple indexes #2486

Web app and Rest API: Support for multiple indexes #2486

Conversation

16BitNarwhal commented May 5, 2024

codecov bot commented May 5, 2024 • edited Loading

Codecov Report

lintool commented May 5, 2024

16BitNarwhal commented May 6, 2024

lintool commented May 6, 2024

lintool May 12, 2024

Choose a reason for hiding this comment

lintool May 12, 2024

Choose a reason for hiding this comment

lintool May 12, 2024

Choose a reason for hiding this comment

codecov bot commented May 5, 2024 •

edited

Loading