Failed to load model from disk

[thomenedis-netizen]

Description

Hello,
Thanks a lot for your great work on eds-pseudo and eds-nlp.
I am trying to run eds-pseudo on an instance with default ephemeral storage.
I succeed in loading "AP-HP/eds-pseudo-public" model from HuggingFace the first time with:

import edsnlp
import edsnlp.pipes as eds

nlp = edsnlp.load("AP-HP/eds-pseudo-public")

But, to avoid having to load it every time, I want to save it on disk (on a persistent path) with:
nlp.to_disk(path)

But, when I start again the instance, I can't load back the model:

nlp = edsnlp.load(path)
RegistryError: Can't find 'eds_pseudo.simple_rules' in registry edsnlp -> factories. Available names: SOFA, accents, attribute_ruler, beam_ner, beam_parser, charlson, consultation_dates, contextual-matcher, dates, doc_cleaner, eds.AIDS, eds.CKD, eds.COPD, eds.SOFA, eds.TNM, eds.accents, eds.adicap, eds.aids, eds.alcohol, eds.biaffine_dep_parser, eds.cerebrovascular_accident, eds.charlson, eds.cim10, eds.ckd, eds.congestive_heart_failure, eds.conll_dict2doc, eds.connective_tissue_disease, eds.consultation_dates, eds.contextual-matcher, eds.contextual_matcher, eds.copd, eds.covid, eds.dates, eds.dementia, eds.diabetes, eds.drugs, eds.elston-ellis, eds.elston_ellis, eds.elstonellis, eds.emergency.ccmu, eds.emergency.gemsa, eds.emergency.priority, eds.emergency_ccmu, eds.emergency_gemsa, eds.emergency_priority, eds.endlines, eds.ents_doc2dict, eds.explode, eds.extractive_qa, eds.family, eds.hemiplegia, eds.history, eds.hypothesis, eds.leukemia, eds.liver_disease, eds.llm_markup_extractor, eds.lymphoma, eds.markup_to_doc, eds.matcher, eds.measurements, eds.measures, eds.myocardial_infarction, eds.negation, eds.ner_crf, eds.nested_ner, eds.normalizer, eds.omop_dict2doc, eds.omop_doc2dict, eds.peptic_ulcer_disease, eds.peripheral_vascular_disease, eds.pollution, eds.quantities, eds.quotes, eds.reason, eds.remove-lowercase, eds.remove_lowercase, eds.reported_speech, eds.score, eds.sections, eds.sentences, eds.sofa, eds.solid_tumor, eds.spaces, eds.span_classifier, eds.span_linker, eds.span_pooler, eds.span_qualifier, eds.split, eds.standoff_dict2doc, eds.standoff_doc2dict, eds.suicide_attempt, eds.tables, eds.terminology, eds.text_cnn, eds.tnm, eds.tobacco, eds.transformer, eds.umls, emergency.ccmu, emergency.gemsa, emergency.priority, endlines, entity_linker, entity_ruler, family, fr.lemmatizer, future_entity_ruler, hypothesis, lemmatizer, matcher, merge_entities, merge_noun_chunks, merge_subtokens, morphologizer, negation, ner, normalizer, parser, pollution, quotes, reason, remove-lowercase, reported_speech, rspeech, score, sections, sentences, sentencizer, senter, spaces, span_finder, span_ruler, spancat, spancat_singlelabel, tables, tagger, terminology, textcat, textcat_multilabel, tok2vec, token_splitter, trainable_lemmatizer

Sorry, for this basic question.
Is there anything I miss to register back eds-pseudo?
Thanks a lot,

How to reproduce the bug

Your Environment

• Operating System: Ubuntu 24.04.3 LTS
• Python Version Used: 3.12
• EDS-Pseudo Version Used: 0.4.0
• Environment Information:

[percevalw]

Hi, @thomenedis-netizen thank you for the bug report ! Indeed I had no planned for this.

The python code required to run some of the rule-based components of the eds-pseudo-public model is located inside the huggingface model archive: https://huggingface.co/AP-HP/eds-pseudo-public/tree/main > eds_pseudo. When you load it with AP-HP/eds-pseudo-public, it does a bit more than loading the weights, as it temporarily installs the "eds_pseudo" package located in the hf archive, which makes some eds-pseudo specific components (e.g. 'eds_pseudo.simple_rules') available.
When you save this same model, it only save its config file and weights, but not the code required to run it, so you can't indeed load it back.

I'll give this some more thoughtn but in the meantime you have two options:

1. drop the rule-based components (I haven't tested this)

   edsnlp.load(path, exclude=["simple-rules", "addresses", "context", "clean", "merge", "dates-normalizer"])

   but you'll lose some features like date normalization, rule-based detection and some PII might end up in doc.spans instead of doc.ents

2. Install the model as a package (see here)

   # Or install it once as a package
   !pip install git+https://huggingface.co/AP-HP/eds-pseudo-public
   
   # and import it as a module
   import eds_pseudo_public
   
   nlp = eds_pseudo_public.load()  # or edsnlp.load("eds_pseudo_public")

Do any of these solutions work for you ?

[thomenedis-netizen]

Hi @percevalw
Thanks for your quick feedback and alternative proposal.
For some internal reasons, I am currently blocked when git cloning https://huggingface.co/AP-HP/eds-pseudo-public
I will try to solve this and let you know if the second option solves my pb.
Thanks