updates for solr based search

[fdefalco]

No description provided.

[anthonysena]

This issue aims to address #580

[anthonysena]

Setup notes updated in #1091 (comment)

[anthonysena]

A few other notes after discussing with @fdefalco:

• We should provide some guidance on how to run SOLR in a durable process (i.e. Windows Service) if possible.
• It is desirable to expose the SOLR settings through the /info endpoint. The elements would be:
  • SOLR enabled
  • List of the cores available
• The vocabulary search endpoint needs to be backwards compatible - this branch makes a breaking change to the endpoint.
• Add the SOLR resources that are mentioned earlier in this PR to the \resources folder for reference.

Nice to haves:

• Expose the InitializeFullTextIndexCache through a secured end-point to enable a refresh of SOLR cores without the need to restart WebAPI
• Provide some admin functionality for building cores via Atlas/WebAPI

[anthonysena]

@fdefalco - this is back to you for review. I've addressed the notes mentioned in #1091 (comment) as well as feedback from @pavgra regarding the use of solrj, IoC approach and change to the configuration settings. From my side, this is working well as I've tested it with SOLR enabled/disabled.

We can look at creating a new issue for Atlas 3.0 for the "nice to haves" mentioned above; let me know if you agree with that.

Still outstanding is:

• Guidance on how to setup & operate SOLR in a durable process - Tomcat should be suitable for this purpose from what I've been reading.

[anthonysena]

Updating the default query behavior to use the SOLR ComplexPhraseQueryParser: https://lucene.apache.org/solr/guide/6_6/other-parsers.html#OtherParsers-ComplexPhraseQueryParser. This will allow for searches that are closer in behavior to the SQL Wildcard search.

[anthonysena]

Handles unicode characters: https://lucene.apache.org/solr/guide/6_6/filter-descriptions.html

[anthonysena]

Final Setup Notes for Wiki

These instructions will focus on Windows since Solr deployment on Linux is covered on Solr install guide.

Install the Solr Windows Service

We will use Apache Procrun to wrap Solr 8.3.1 in a Windows service to ensure we can control start up/shutdown like other services. To do this, follow the following steps

• Downloaded and install Apache SOLR 8.3.1 from https://lucene.apache.org/solr/downloads.html. For this guide, we will use E:\solr\solr-8.3.1 as the install location.
• Download Apache Commons Procrun for Windows (Procrun): http://www.apache.org/dist/commons/daemon/binaries/windows/. Extract the install to E:\temp\procrun.
• Copy prunsrv.exe from the procrun directory E:\solr\solr-8.3.1\bin
• Create a new file called service.bat in E:\solr\solr-8.3.1\bin and copy in the following text:

@echo off
set SERVICE_NAME=Solr
set SERVICE_HOME=E:\solr\solr-8.3.1
set PR_INSTALL=%SERVICE_HOME%\bin\prunsrv.exe

@REM Service Log Configuration
set PR_LOGPREFIX=%SERVICE_NAME%
set PR_LOGPATH=%SERVICE_HOME%\logs
set PR_STDOUTPUT=auto
set PR_STDERROR=auto
set PR_LOGLEVEL=Debug

set PR_STARTUP=auto
set PR_STARTMODE=exe
set PR_STARTIMAGE=%SERVICE_HOME%\bin\solr.cmd
set PR_STARTPARAMS=start

@REM Shutdown Configuration
set PR_STOPMODE=exe
set PR_STOPIMAGE=%SERVICE_HOME%\bin\solr.cmd
set PR_STOPPARAMS=stop;-all

%PR_INSTALL% //IS/%SERVICE_NAME% ^
  --Description="Apache Solr 8.3.1" ^
  --DisplayName="%SERVICE_NAME%" ^
  --Install="%PR_INSTALL%" ^
  --Startup="%PR_STARTUP%" ^
  --LogPath="%PR_LOGPATH%" ^
  --LogPrefix="%PR_LOGPREFIX%" ^
  --LogLevel="%PR_LOGLEVEL%" ^
  --StdOutput="%PR_STDOUTPUT%" ^
  --StdError="%PR_STDERROR%" ^
  --StartMode="%PR_STARTMODE%" ^
  --StartImage="%PR_STARTIMAGE%" ^
  ++StartParams="%PR_STARTPARAMS%" ^
  --StopMode="%PR_STOPMODE%" ^
  --StopImage="%PR_STOPIMAGE%" ^
  ++StopParams="%PR_STOPPARAMS%"

if not errorlevel 1 goto installed
echo Failed to install "%SERVICE_NAME%" service.  Refer to log in %PR_LOGPATH%
exit /B 1

:installed
echo The Service "%SERVICE_NAME%" has been installed
exit /B 0

NOTE: Adjust the SERVICE_HOME setting to match your install location.

Run a Windows Command Prompt in Administrator mode and then run E:\solr\solr-8.3.1\bin\service.bat. The service will be installed in the list of Windows Services as "Solr". Before moving forward, confirm the service is created but not running.

Creating the Solr core for WebAPI vocabulary search

NOTE: The name of the Solr core used must match the vocabulary version you plan to use in ATLAS & WebAPI with an underscore. For this example, the vocabulary version is "v5.0 17-JUN-19" and the corresponding folder name to hold this vocabulary is "v5.0_17-JUN-19". You should verify your vocabulary by running the following query on the CDM(s) you plan to use with WebAPI:

select vocabulary_version from vocabulary where vocabulary_id = 'None';

If your vocabulary version and core do not match, WebAPI will not find the Solr core and it will continue to use the DB when querying the vocabulary.

Solr core creation

• Verify that the JAR files for your RDMBS are located in E:\solr\solr-8.3.1\server\lib otherwise you will face issues when attempting to build the SOLR core.
• Created 2 directories for the core:
  • E:\solr\solr-8.3.1\server\solr\v5.0_17-JUN-19
  • E:\solr\solr-8.3.1\server\solr\v5.0_17-JUN-19\data
• Copy the contents of WebAPI\src\main\resources\solr into E:\solr\solr-8.3.1\server\solr\v5.0_17-JUN-19. Next make the following edits to the files in E:\solr\solr-8.3.1\server\solr\v5.0_17-JUN-19:
  • data-config.xml: Edit the data source & references in the query to match the database holding your vocabulary
  • core.properties: Edit the name to match the directory: v5.0_17-JUN-19
  • conf\solrconfig.xml: Edit the connection information in the <requestHandler name="/dataimport" class="solr.DataImportHandler"> block to provide it the details to connect the database holding your vocabulary.

Building the Solr core

Next, start up the Solr windows service and verify connect to http://localhost:8983. If there are problems starting up the service, please review the logs found in E:\solr\solr-8.3.1\server\logs.

• From the SOLR Admin screen (http://localhost:8983/solr/#/), build the core by using the 'Core Selector' dropdown in the left-hand menu. Select the v5.0 17-JUN-19 core from the drop down, and then in the sub-menu that appears, I selected Dataimport and then used the 'Execute' button. The service will build the core from the concepts per the query in the data-config.xml
• Once the execution of the core indexing is complete, you can use the Solr "Query" tool under the core sub-menu to make sure the core is working properly before moving to WebAPI. A sample query you can use is: query:metformin

WebAPI Configuration

In the settings.xml for WebAPI, add the following XML to your profile:

  <solr.endpoint>http://localhost:8983/solr</solr.endpoint>

Recompile and deploy WebAPI.war. Once deployed, verify that the SOLR service is available for search by going to the endpoint WebAPI/info. The output will look similar to this:

{
	"version": "2.8.0",
	"buildInfo": {
		"artifactVersion": "WebAPI 2.8.0-SNAPSHOT",
		"build": "NA",
		"timestamp": "Thu Dec 12 16:53:22 UTC 2019"
	},
	"configuration": {
		"security": {
			"enabled": true
		},
		"vocabulary": {
			"cores": [
				"v5.0 17-JUN-19"
			],
			"solrEnabled": true
		},
		"person": {
			"viewDatesPermitted": false
		},
		"heracles": {
			"smallCellCount": "5"
		}
	}
}

Note in the JSON above, the vocabulary section shows the Solr is enabled and lists the core(s) that are available for search.

All review items addressed.

[alondhe]

Updated to use SOLR 8.11.1 due to log4j vulnerability

Install the Solr Service on RedHat (tested on v7)

Download the binary and install (sudo is needed)

cd /opt
# Download the binary (https://solr.apache.org/downloads.html) -- v8.11.1 tgz file

tar xzf solr-8.11.1.tgz solr-8.11.1/bin/install_solr_service.sh --strip-components=2
bash ./install_solr_service.sh solr-8.11.1.tgz

Solr core creation

1. Verify that the JAR files for your RDMBS are located in /opt/solr/server/lib otherwise you will face issues when attempting to build the SOLR core.

2. Create 2 directories for the core:

• /var/solr/data/v5.0_12-FEB-21
• /var/solr/data/v5.0_12-FEB-21/data

3. Copy the contents of WebAPI/src/main/resources/solr into /var/solr/data/v5.0_12-FEB-21.

4. Next make the following edits to the files in /var/solr/data/v5.0_12-FEB-21:

• data-config.xml: Edit the data source & references in the query to match the database holding your vocabulary
• core.properties: Edit the name to match the directory: v5.0_12-FEB-21, remove the "conf\\" prefix from config and schema values.
• conf\solrconfig.xml: Edit the connection information in the block to provide it the details to connect the database holding your vocabulary.
  • if using Spark, add this to this /dataimport block: <str name="autoCommit">true</str>

Building the Solr core

Next, start up the Solr service (systemctl start solr) and verify connect to http://localhost:8983. If there are problems starting up the service, please review the logs found in /var/solr/logs.

• From the SOLR Admin screen (http://localhost:8983/solr/#/), build the core by using the 'Core Selector' dropdown in the left-hand menu. Select the v5.0_12-FEB-21 core from the drop down, and then in the sub-menu that appears, I selected Dataimport and then used the 'Execute' button. The service will build the core from the concepts per the query in the data-config.xml
• Once the execution of the core indexing is complete, you can use the Solr "Query" tool under the core sub-menu to make sure the core is working properly before moving to WebAPI. A sample query you can use is to enter this in the q field: query:metformin

Then, follow the rest of the WebAPI instructions in the Windows instructions above.

[alondhe]

@anthonysena - the query for the core only uses the concept table, should it also union STCM records? I know STCM is deprecated (ish), but there are site-specific mappings we (and I believe others) use there.

[alondhe]

Updated to use SOLR 8.11.1 due to log4j vulnerability

Install SOLR using Docker

On Windows/Mac:

Download Docker desktop app (https://www.docker.com/products/docker-desktop).

On Debian/Ubuntu Linux:

sudo apt-get install docker

On RHEL:

sudo yum install docker

Solr core creation

1. Create a directory to store the Dockerfile and configuration files, this directory is known as the docker build context folder. Navigate to it in your command line interface.

2. Create a file named "Dockerfile" in this build context folder and use this as the content:

FROM solr:8.11.1

# argument variables to define
ARG vocabulary_version
ARG jdbc_file_name

# copy the solr configset from WebAPI
COPY --chown=solr /WebAPI/src/main/resources/solr /var/solr/data/$vocabulary_version

# copy your JDBC file
COPY --chown=solr $jdbc_file_name /opt/solr-8.11.1/server/lib/$jdbc_file_name

3. Clone the WebAPI Git repo (using your desired commit or release) into the docker build context folder:

git clone https://github.com/OHDSI/WebAPI.git

4. Copy your JDBC jar file (for connecting to your vocabulary's database platform) into the build context folder.

5. Next make the following edits to the files in /WebAPI/src/main/resources/solr:

• data-config.xml: Edit the data source & references in the query to match the database holding your vocabulary
• core.properties: Edit the name to match the vocabulary version (e.g. v5.0_20-MAY-21), remove the "conf\\" prefix from config and schema values.
• conf\solrconfig.xml: Edit the connection information in the block to provide it the details to connect the database holding your vocabulary.
  • if using Spark, add this to this /dataimport block: <str name="autoCommit">true</str>

Build/create/start the docker container

5. Run the container build step, specifying the vocabulary version and the name of the JDBC file needed for connecting to your database platform:

docker build --build-arg vocabulary_version=v5.0_20-MAY-21 --build-arg jdbc_file_name=SparkJDBC41.jar --no-cache -t solr .

6. Run the create step, which will then create the container:

docker create --restart=always --name=solr -p 8983:8983 -t solr

7. Start the container:

docker start solr

8. Test the container has started by going to http://localhost:8983/solr/#/ (substitute the server name for localhost)

Building the Solr core

• From the SOLR Admin screen (http://localhost:8983/solr/#/), build the core by using the 'Core Selector' dropdown in the left-hand menu. Select the vocabulary core from the drop down, and then in the sub-menu that appears, select Dataimport and then used the 'Execute' button. The service will build the core from the concepts per the query in the data-config.xml
• Once the execution of the core indexing is complete, you can use the Solr "Query" tool under the core sub-menu to make sure the core is working properly before moving to WebAPI. A sample query you can use is to enter this in the q field: query:metformin

Then, follow the rest of the WebAPI configuration instructions in the Windows instructions above.

[alondhe]

With Atlas 2.12 adding invalid_start_date and invalid_end_date, the SOLR core creation now needs to handle these fields.

Edit: never mind, handled already