Error on indexing with unclear error messages because no available reference sets for selected locale

[wardle]

As raised by @ppazos - error on indexing with latest International edition (Sept 2023):

$ java -jar hermes-1.4.1278.jar --db snomed_db index compact
2023-09-18 19:17:42,453 [main] INFO  http://com.eldrix.hermes.core - Indexing... {:root "snomed_db"}
2023-09-18 19:17:42,456 [main] INFO  http://com.eldrix.hermes.core - Building component index
2023-09-18 19:17:42,800 [main] INFO  http://com.eldrix.hermes.core - Building search index
2023-09-18 19:17:43,046 [main] INFO  http://com.eldrix.hermes.core - Building members index
2023-09-18 19:17:43,108 [main] INFO  http://com.eldrix.hermes.core - Indexing... complete
2023-09-18 19:17:43,149 [main] ERROR http://com.eldrix.hermes.cmd.core - Uncaught exception on "main"
clojure.lang.ExceptionInfo: cannot use requested fallback language priority list with current installed language reference sets
at http://com.eldrix.hermes.impl.language$make_match_fn.invokeStatic(language.clj:213)
at http://com.eldrix.hermes.core$open.invokeStatic(core.clj:1241)
at http://com.eldrix.hermes.cmd.core$build_index.invokeStatic(core.clj:63)
at http://com.eldrix.hermes.cmd.core$build_index.invoke(core.clj:63)
at http://com.eldrix.hermes.cmd.core$invoke_command.invokeStatic(core.clj:132)
at http://com.eldrix.hermes.cmd.core$_main.invokeStatic(core.clj:160)
at http://com.eldrix.hermes.cmd.core$_main.doInvoke(core.clj:141)
at clojure.lang.RestFn.applyTo(http://RestFn.java:137)
at http://com.eldrix.hermes.cmd.core.main(Unknown Source)

But when that index is run on JVM, the error message is better:

Exception in thread "Thread-10" clojure.lang.ExceptionInfo: cannot use requested fallback language priority list with current installed language reference sets {:requested "en-US", :installed {}}
at http://com.eldrix.hermes.impl.language$make_match_fn.invokeStatic(language.clj:214)
at http://com.eldrix.hermes.impl.language$make_match_fn.invoke(language.clj:194)
at http://com.eldrix.hermes.core$open.invokeStatic(core.clj:1247)
at http://com.eldrix.hermes.core$open.invoke(core.clj:1227)
at http://com.eldrix.hermes.core$open.invokeStatic(core.clj:1233)
at http://com.eldrix.hermes.core$open.invoke(core.clj:1227)
at clojure.lang.Var.invoke(http://Var.java:384)
at http://com.eldrix.hermes.client.Hermes.<init>(http://Hermes.java:89)
at http://com.eldrix.hermes.client.Hermes.openLocal(https://t.co/8Q9fOUyAJA)
at http://com.eldrix.hermes.client.Hermes$openLocal.call(Unknown Source)

[wardle]

Firstly, the error reporting should be consistent and more detailed, so that it explains what has happened, and provides better diagnostic information. Fixed in 17fd09a so that Hermes will report the same information in the Exception and log at the command line, and provide advice on how to address.

[wardle]

In particular, it is very confusing that Hermes reports "Uncaught exception" as that implies that something untoward has happened, when in fact this is appropriate behaviour. Fixed in 7112aba

[wardle]

Secondly, I cannot understand why Hermes isn't recognising that the en-US locale is available from what can be installed in the Sept 23 edition - I have tried to reproduce this and it works fine.

For example, Hermes can download automatically from the International release centre:

$ clj -M:run --db snomed-intl-2023-09 install ihtsdo.mlds/167 --password mlds-password.txt --username mark@wardle.org
2023-09-19 18:31:12,426 [main] INFO  com.eldrix.hermes.download - installing distribution {:distribution "ihtsdo.mlds/167", :params {:db "snomed-intl-2023-09", :password "mlds-password.txt", :username "mark@wardle.org"}}
2023-09-19 18:31:12,709 [main] INFO  com.eldrix.hermes.download - downloading MLDS package {:id "ihtsdo.mlds/167", :version 1030283, :release-date "2023-09-01"}
2023-09-19 18:31:12,709 [main] INFO  com.eldrix.hermes.download - downloading MLDS release file {:url "/api/releasePackages/167/releaseVersions/1030283/releaseFiles/1030360/download", :file "SnomedCT_InternationalRF2_PRODUCTION_20230901T120000Z.zip"}
2023-09-19 18:36:21,867 [main] INFO  com.eldrix.hermes.core - importing 1 distribution(s) from "/var/folders/55/4p0b7_tx5js8qdt8bg6cvsqc0000gn/T/trud13328207838337688280"
2023-09-19 18:36:21,868 [main] INFO  com.eldrix.hermes.core - distribution:  {:name "SnomedCT_InternationalRF2_PRODUCTION_20230901T120000Z", :effectiveTime #object[java.time.LocalDate 0x2536d8aa "2023-09-01"]}
2023-09-19 18:36:21,868 [main] INFO  com.eldrix.hermes.core - license:  "© 2023 International Health Terminology Standards Development Organisation 2002-2023.  All rights reserved.  SNOMED CT® was originally created by the College of American Pathologists.  'SNOMED' and 'SNOMED CT' are registered trademarks of International Health Terminology Standards Development Organisation, trading as SNOMED International.  SNOMED CT has been created by combining SNOMED RT and a computer based nomenclature and classification known as Clinical Terms Version 3, formerly known as Read Codes Version 3, which was created on behalf of the UK Department of Health and is Crown copyright.  This document forms part of the International Edition release of SNOMED CT® distributed by SNOMED International, which is subject to the SNOMED CT® Affiliate License, details of which may be found at  https://www.snomed.org/snomed-ct/get-snomed."
2023-09-19 18:36:22,212 [async-thread-macro-1] INFO  com.eldrix.hermes.importer - Processing:  "der2_cciRefset_RefsetDescriptorSnapshot_INT_20230901.txt"  type:  "RefsetDescriptorRefset"
2023-09-19 18:36:22,214 [async-thread-macro-1] INFO  com.eldrix.hermes.importer - Processing:  "sct2_Relationship_Snapshot_INT_20230901.txt"  type:  "Relationship"
2023-09-19 18:36:26,396 [async-thread-macro-1] INFO  com.eldrix.hermes.importer - Processing:  "sct2_Description_Snapshot-en_INT_20230901.txt"  type:  "Description"
2023-09-19 18:36:29,492 [async-thread-macro-1] INFO  com.eldrix.hermes.importer - Processing:  "sct2_Concept_Snapshot_INT_20230901.txt"  type:  "Concept"
2023-09-19 18:36:32,446 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_Refset_SimpleSnapshot_INT_20230901.txt"  type:  "SimpleRefset"
2023-09-19 18:36:32,484 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_cRefset_AssociationSnapshot_INT_20230901.txt"  type:  "AssociationRefset"
2023-09-19 18:36:32,806 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_cRefset_AttributeValueSnapshot_INT_20230901.txt"  type:  "AttributeValueRefset"
2023-09-19 18:36:33,937 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_cRefset_LanguageSnapshot-en_INT_20230901.txt"  type:  "LanguageRefset"
2023-09-19 18:36:38,926 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_sRefset_SimpleMapSnapshot_INT_20230901.txt"  type:  "SimpleMapRefset"
2023-09-19 18:36:39,809 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_iisssccRefset_ExtendedMapSnapshot_INT_20230901.txt"  type:  "ExtendedMapRefset"
2023-09-19 18:36:40,252 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_cissccRefset_MRCMAttributeDomainSnapshot_INT_20230901.txt"  type:  "MRCMAttributeDomainRefset"
2023-09-19 18:36:40,260 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_ssccRefset_MRCMAttributeRangeSnapshot_INT_20230901.txt"  type:  "MRCMAttributeRangeRefset"
2023-09-19 18:36:40,269 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_ssRefset_ModuleDependencySnapshot_INT_20230901.txt"  type:  "ModuleDependencyRefset"
2023-09-19 18:36:40,278 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_sssssssRefset_MRCMDomainSnapshot_INT_20230901.txt"  type:  "MRCMDomainRefset"
2023-09-19 18:36:40,286 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "der2_cRefset_MRCMModuleScopeSnapshot_INT_20230901.txt"  type:  "MRCMModuleScopeRefset"
2023-09-19 18:36:40,295 [async-thread-macro-7] INFO  com.eldrix.hermes.importer - Processing:  "sct2_RelationshipConcreteValues_Snapshot_INT_20230901.txt"  type:  "RelationshipConcreteValues"

We can then index and compact:

$ clj -M:run --db snomed-intl-2023-09 index                                                                          
2023-09-19 18:37:56,694 [main] INFO  com.eldrix.hermes.core - Indexing... {:root "snomed-intl-2023-09"}
2023-09-19 18:37:56,695 [main] INFO  com.eldrix.hermes.core - Building component index
2023-09-19 18:38:06,544 [main] INFO  com.eldrix.hermes.core - Building search index
2023-09-19 18:38:40,662 [main] INFO  com.eldrix.hermes.core - Building members index
2023-09-19 18:39:27,090 [main] INFO  com.eldrix.hermes.core - Indexing... complete

$ clj -M:run --db snomed-intl-2023-09 compact
2023-09-19 18:39:43,055 [main] INFO  com.eldrix.hermes.core - Compacting database at  "snomed-intl-2023-09" ...
2023-09-19 18:39:47,055 [main] INFO  com.eldrix.hermes.core - Compacting database... complete

Import and index and compact proceed with no error.

$ clj -M:run --db snomed-intl-2023-09 status 
{:releases
 ("SNOMED Clinical Terms version: 20230901 [R] (September 2023 Release)"),
 :locales ("en-GB" "en-US"),
 :components
 {:concepts 506462,
  :descriptions 1608259,
  :relationships 3336318,
  :concrete-values 34822,
  :refsets 26,
  :refset-items 4784009,
  :indices
  {:descriptions-concept 1608259,
   :concept-parent-relationships 1242079,
   :concept-child-relationships 1242079,
   :component-refsets 4045941,
   :associations 171701,
   :descriptions-search 1608259,
   :members-search 4784009}}}

And since the commits above, if we ask for a locale not supported by the installed language reference sets, then we get much better information:

$ clj -M:run --db snomed-intl-2023-09 serve --locale fr
2023-09-19 18:43:55,282 [main] ERROR com.eldrix.hermes.impl.language - No language reference set installed matching requested fallback (default) locale. {:requested "fr", :available ("en-GB" "en-US"), :unrecognized (), :installed #{900000000000509007 900000000000508004}}
2023-09-19 18:43:55,283 [main] ERROR com.eldrix.hermes.impl.language - Explicitly choose a default locale or install additional distributions
Execution error (ExceptionInfo) at com.eldrix.hermes.impl.language/make-match-fn (language.clj:238).
No language reference set installed matching requested fallback (default) locale.

Full report at:
/var/folders/55/4p0b7_tx5js8qdt8bg6cvsqc0000gn/T/clojure-1184960818726131772.edn

[wardle]

But that does leave two unanswered questions.

1. Why did it fail for @ppazos ? His system default locale is en-US and yet his installed datafiles seem to have no reference sets. Can you reproduce this with HEAD ?
2. Should Hermes fallback to a different default locale than the system locale? For example, if a user explicitly specifies a fallback default locale on opening a datafile, then if that request cannot be satisfied, it is, I think, appropriate to throw an error. They've likely made a mistake. But, if we use the system default locale, then perhaps Hermes could determine a fallback based on installed locales? My preference has been to get the user to be explicit, because this is less likely to result in surprising behaviour, but I can be persuaded of a smarter fallback mechanism if the system locale cannot be supported by the range of reference sets installed in a given datafile.

[wardle]

See release v1.4.1292 - available as uberjar and as a library. If you can try with this release, it will provide much better diagnostic information. As I said above, I downloaded the same Sept 23 Int'l release and had no problems at all.