Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VUFIND-1630] Alphabrowse: new normalizer for titles based on SolrMarc titleSortLower #3024

Merged
Merged
Show file tree
Hide file tree
Changes from 7 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
920ba46
[VUFIND-1630] Alphabrowse: new normalizer for titles based on SolrMar…
damien-git Aug 4, 2023
862a820
[VUFIND-1630] SOLR_JAR_PATH env + classpath fix
damien-git Aug 10, 2023
230f790
[VUFIND-1630] Classpath optimization
damien-git Aug 10, 2023
6eac2f8
[VUFIND-1630] Removed duplicate solrmarc_core.jar, updated browse-han…
damien-git Aug 10, 2023
44c8488
[VUFIND-1630] Updated browse-indexing.jar
damien-git Aug 14, 2023
b2e92f5
[VUFIND-1630] Added check for single solrmarc jar
damien-git Aug 18, 2023
81110e0
Merge branch 'dev' into fix_alphabrowse_title_normalizer
demiankatz Sep 7, 2023
0c19788
Add titleSortLower helper to XSLT code.
demiankatz Sep 7, 2023
81ba97c
Use new helper in XSLT examples.
demiankatz Sep 7, 2023
b915e4e
Fix whitespace glitch.
demiankatz Sep 7, 2023
d641363
Add another test case.
demiankatz Sep 7, 2023
0acbc25
[VUFIND-1630] Removed Log4j classpath as import/lib/reload4j-*.jar wi…
damien-git Sep 12, 2023
eca24b9
Merge branch 'dev' into fix_alphabrowse_title_normalizer
demiankatz Sep 12, 2023
beff9a5
Add cleanData logic.
demiankatz Sep 12, 2023
6818df3
php-cs-fixer.
demiankatz Sep 12, 2023
b9e80dd
Fix phpstan issues.
demiankatz Sep 12, 2023
d8fa924
Further phpstan refinement.
demiankatz Sep 12, 2023
b5bb9ea
More regex fixes.
demiankatz Sep 12, 2023
e895ce0
Add SOLR_JAR_PATH support and comment.
demiankatz Sep 13, 2023
61db918
Bare minimum SOLRMARC_CLASSPATH support.
demiankatz Sep 13, 2023
72c220d
Add counting/validation logic for jar detection.
demiankatz Sep 13, 2023
2f715f3
Merge branch 'dev' into fix_alphabrowse_title_normalizer
demiankatz Sep 14, 2023
2cacb9b
Expand/simplify AlphabrowseTest.
demiankatz Sep 14, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Binary file modified import/browse-indexing.jar
Binary file not shown.
2 changes: 2 additions & 0 deletions import/marc.properties
Original file line number Diff line number Diff line change
Expand Up @@ -63,6 +63,8 @@ title_auth = 245ab, first
title_alt = 100t:130adfgklnpst:240a:246abnp:505t:700t:710t:711t:730adfgklnpst:740a
title_old = 780ast
title_new = 785ast
# If titleSortLower is changed in title_sort, the normalizer in solr/vufind/biblio/conf/solrconfig.xml and
# index-alphabetic-browse.sh should be changed accordingly.
title_sort = 245abkp,titleSortLower,first
series = 440ap:800abcdfpqt:830ap
series2 = 490a
Expand Down
18 changes: 16 additions & 2 deletions index-alphabetic-browse.sh
Original file line number Diff line number Diff line change
Expand Up @@ -31,11 +31,24 @@ then
SOLR_HOME="$VUFIND_HOME/solr/vufind"
fi

# This would be /opt/bitnami/solr on a solr docker container
demiankatz marked this conversation as resolved.
Show resolved Hide resolved
if [ -z "$SOLR_JAR_PATH" ]
then
SOLR_JAR_PATH="${SOLR_HOME}/../vendor"
fi

set -e
set -x

cd "`dirname $0`/import"
CLASSPATH="browse-indexing.jar:${VUFIND_HOME}/import/lib/*:${SOLR_HOME}/jars/*:${SOLR_HOME}/../vendor/modules/analysis-extras/lib/*:${SOLR_HOME}/../vendor/server/solr-webapp/webapp/WEB-INF/lib/*"
LOG4J_CLASSPATH=$(find "${SOLR_JAR_PATH}/server/lib/ext" -name 'log4j*.jar' -printf '%p:' | sed 's/:$//')
demiankatz marked this conversation as resolved.
Show resolved Hide resolved
SOLRMARC_CLASSPATH=$(echo solrmarc_core*.jar)
if [[ `wc -w <<<"$SOLRMARC_CLASSPATH"` -gt 1 ]]
then
echo "Error: more than one solrmarc_core*.jar in import/; exiting."
exit 1
fi
CLASSPATH="browse-indexing.jar:${SOLRMARC_CLASSPATH}:${VUFIND_HOME}/import/lib/*:${SOLR_HOME}/jars/*:${SOLR_JAR_PATH}/modules/analysis-extras/lib/*:${SOLR_JAR_PATH}/server/solr-webapp/webapp/WEB-INF/lib/*:${LOG4J_CLASSPATH}"

# make index work with replicated index
# current index is stored in the last line of index.properties
Expand Down Expand Up @@ -87,8 +100,9 @@ function build_browse
mv "${browse}_browse.db" "$index_dir/${browse}_browse.db-updated"
touch "$index_dir/${browse}_browse.db-ready"
}
# These parameters should match the ones in solr/vufind/biblio/conf/solrconfig.xml - BrowseRequestHandler
build_browse "hierarchy" "hierarchy_browse"
build_browse "title" "title_fullStr" 1 "-Dbibleech=StoredFieldLeech -Dsortfield=title_sort -Dvaluefield=title_fullStr"
build_browse "title" "title_fullStr" 1 "-Dbibleech=StoredFieldLeech -Dsortfield=title_sort -Dvaluefield=title_fullStr -Dbrowse.normalizer=org.vufind.util.TitleNormalizer"
build_browse "topic" "topic_browse"
build_browse "author" "author_browse"
build_browse "lcc" "callnumber-raw" 1 "-Dbrowse.normalizer=org.vufind.util.LCCallNormalizer"
Expand Down
3 changes: 3 additions & 0 deletions solr/vufind/biblio/conf/solrconfig.xml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,7 @@
<dataDir>${solr.solr.home:./solr}/biblio</dataDir>

<lib dir="../../../import/lib" regex="marc4j.*\.jar" />
<lib dir="../../../import" regex="solrmarc_core.*\.jar" />
<lib dir="../../vendor/modules/analysis-extras/lib" regex=".*\.jar" />
<lib dir="../jars" regex=".*\.jar" />

Expand Down Expand Up @@ -437,6 +438,7 @@

<requestHandler name="/browse" class="org.vufind.solr.handler.BrowseRequestHandler">
<!-- These definitions should match the field names used in the authority index. -->
<!-- The parameters should match the ones given to build_browse in index-alphabetic-browse.sh -->
<str name="preferredHeadingField">heading</str>
<str name="useInsteadHeadingField">use_for</str>
<str name="seeAlsoHeadingField">see_also</str>
Expand All @@ -457,6 +459,7 @@

<lst name="title">
<str name="DBpath">${solr.solr.home:./solr}/alphabetical_browse/title_browse.db</str>
<str name="normalizer">org.vufind.util.TitleNormalizer</str>
<str name="field">title_fullStr</str>
</lst>

Expand Down
Binary file modified solr/vufind/jars/browse-handler.jar
Binary file not shown.