Migration workflow

Migration workflow steps

v1.0.5 → v1.1.0

Run the steps in order.

1. Put the new index template

Shard and replica counts moved into a registered template. Do not use invenio index init: it also creates the indexes and fails on an existing instance. Via uv run invenio shell:

from invenio_search import current_search
for name, response in current_search.put_templates():
    print(name, response)

No index has to be rebuilt.

2. Add the concept association fields to the live mappings

uv run invenio rero es index update-mapping

Only if that reports an error on _association_level, rebuild instead:

uv run invenio rero es index rebuild concepts_idref
uv run invenio rero es index rebuild concepts_gnd
uv run invenio rero es index rebuild concepts_rero

3. Upgrade alembic

uv run invenio alembic upgrade

Revision dcdc05a29568 changes nothing in the database. It only checks that the concept indexes map the association fields, and leaves itself unstamped when one does not, so it can be run again after step 2.

4. Rebuild the concept associations

uv run invenio utils rebuild-concept-association

Reindexes the three concept sources, rebuilds every concept MEF record on the normalised BNF number, drops the ones left without entity. Long-running: start it detached. Each record is committed on its own; on failure it prints the pid and the command to resume, for instance:

uv run invenio utils rebuild-concept-association -t cidref -t cognd -t corero \
  --no-reindex --from-pid 175232768

--no-reindex is only correct once the reindex has run through.

5. Re-harvest the IdRef and then the GND concepts

uv run invenio oaiharvester harvestname -n concepts.idref -f 1990-01-01 -5
uv run invenio oaiharvester harvestname -n concepts.gnd -f 1990-01-01 -5

Only a harvest runs a transformation again; a reindex works from the stored JSON. IdRef first: it rewrites _association_identifier, the field both sides look each other up by, so the GND harvest then searches a correct index.

Both have to run after step 4, and both are long-running.

6. Actions in the harvest log

action	meaning
discard	deleted at the source and never held here; nothing is created
delete	held here, no longer named by the source; deleted with its MEF record

Expect many discards: the IdRef window 2003-12-19 alone reports create=3 discard=2562.

delete clears the TAG: <tag> NOT FOUND records the old transformations invented, so those need no separate cleanup. A record with a relation_pid is never deleted, and keeps the heading already stored.

These rules live in create_or_update, so the scheduled agent and place harvests report them too.

7. Delete the tombstones already stored

A tombstone that carries a real heading comes back uptodate and stays. On mef.test: 1937 cidref, 2525 aidref, 5601 aggnd, 46 pidref. delete-deleted skips those still used in RERO ILS and those with a relation_pid.

python correct_mef.py utils delete-deleted -l delete_deleted.log
python correct_mef.py utils delete-deleted -l delete_deleted.log --commit

One reporting run each for what the old code stranded:

python correct_mef.py utils delete-without-access-point
python correct_mef.py utils delete-empty-deleted-mef -a

Neither is routine maintenance any more.

8. Verify

uv run invenio rero es index update-mapping

Via uv run invenio shell, every line should report 0:

from rero_mef.agents import AgentMefRecord
from rero_mef.concepts import ConceptMefRecord
from rero_mef.places import PlaceMefRecord
from rero_mef.utils import get_entity_search_class

for pid_type in ("aidref", "aggnd", "agrero", "cidref", "cognd", "corero", "pidref", "plgnd"):
    if search := get_entity_search_class(pid_type):
        missing = search().query("bool", must_not=[{"exists": {"field": "authorized_access_point"}}]).count()
        invented = search().query("match_phrase", authorized_access_point="NOT FOUND").count()
        print(f"{pid_type}: no access point {missing}, invented {invented}")

for mef_cls in (AgentMefRecord, ConceptMefRecord, PlaceMefRecord):
    print(f"{mef_cls.__name__} without any entity: {len(list(mef_cls.get_all_pids_without_entities_and_viaf()))}")

invented counts a phrase, so two real GND agents named "File Not Found" and "Error 404 Band not Found. Musikgruppe" show up as well.

from rero_mef.concepts import ConceptMefRecord

linked = ConceptMefRecord.search().filter("exists", field="idref").filter("exists", field="gnd")
print(f"concept MEF: {ConceptMefRecord.count()}, with an idref and a gnd: {linked.count()}")

Nothing to do for

The rest of the release is development-only: docker and test stack, scripts/, generated certificates, lint updates.

INVENIO_CACHE_TYPE=redis became RedisCache, but only in docker-services.yml and the test fixtures. Flask-Caching 2.4 accepts both, so a ConfigMap stating redis keeps working.

v0.17.0 --> v1.0.1

all_mef alias

uv run invenio utils all_mef_alias

change mapping

uv run invenio rero es index update-mapping
uv run invenio rero es index rebuild concepts_idref
uv run invenio rero es index rebuild concepts_gnd
uv run invenio rero es index rebuild concepts_mef
uv run invenio rero es index rebuild places_idref
uv run invenio rero es index rebuild places_gnd
uv run invenio rero es index rebuild places_mef
uv run invenio rero es index rebuild viaf
uv run invenio rero es index update-mapping

Upgrade alembic

uv run invenio alembic upgrade

Fix stale/missing deleted flags on MEF records

DeletedStateExtension (rero_mef/extensions/deleted.py, added in 017580c) propagates a source entity's deleted date onto its MEF aggregate, but only runs on record create/commit. MEF records whose linked source was deleted before v1.0.1 landed were never touched by it, so their deleted field can be stale or missing. Run this once to walk every MEF record, re-run the same propagation logic, and persist+reindex only the ones that actually change:

# uv run invenio shell
from rero_mef.agents.mef.api import AgentMefRecord
from rero_mef.concepts.mef.api import ConceptMefRecord
from rero_mef.places.mef.api import PlaceMefRecord
from rero_mef.extensions.deleted import DeletedStateExtension

ext = DeletedStateExtension()
total = 0
errors = []
for mef_cls in [AgentMefRecord, ConceptMefRecord, PlaceMefRecord]:
    for record in mef_cls.get_all_records():
        before = record.get("deleted")
        try:
            changed = ext._propagate_deleted(record)
        except Exception as exc:
            print(f"{mef_cls.name} {record.pid}: SKIPPED ({exc!r})")
            errors.append((mef_cls.name, record.pid, repr(exc)))
            continue
        if changed:
            print(f"{mef_cls.name} {record.pid}: deleted {before!r} -> {record.get('deleted')!r}")
            try:
                record.commit()
                record.dbcommit(reindex=True)
                total += 1
            except Exception as exc:
                print(f"{mef_cls.name} {record.pid}: SAVE FAILED ({exc!r})")
                errors.append((mef_cls.name, record.pid, repr(exc)))

print(f"\nDone -- {total} MEF record(s) fixed, {len(errors)} skipped.")
if errors:
    print("Skipped (needs manual look — likely a dangling $ref to a purged source entity):")
    for name, pid, exc in errors:
        print(f"  {name} {pid}: {exc}")

VIAF harvesting changed (no script — infra change)

The old VIAF OAI-PMH pull is non-functional as of 58184d0. The new pull-based invenio agents harvest-viaf command needs to be wired into your cron/k8s CronJob outside this repo, or VIAF silently stops updating after the upgrade.

v0.16.0 --> v0.17.0

ConfigMap

• INVENIO_APP_ALLOWED_HOSTS -> INVENIO_TRUSTED_HOSTS
• replace INVENIO_SQLALCHEMY_POOL_RECYCLE by INVENIO_SQLALCHEMY_ENGINE_OPTIONS: '{ "pool_recycle": 360 }'
• set the config in the sqlalchemy container/server by setting PGOPTIONS to -c statement_timeout=30s -c idle_in_transaction_session_timeout=60s
• add PYTHONWARNINGS: ignore

Database

Clean alembic db from invenio_oaiserver

from invenio_db import db
from sqlalchemy import text
db.session.execute(text("DELETE from alembic_version where version_num = '5d25c1981985'"))
db.session.commit()

Upgrade alembic

uv run invenio alembic upgrade

ES

test mapping

uv run invenio rero es index update-mapping

v0.15.0 --> v0.16.0

DB

create new tables

poetry run invenio db create

Mapping

Elasticsearch

poetry run invenio rero es index update-mapping

create ES mapping

poetry run invenio index create  -b rero_mef/places/gnd/mappings/v7/places_gnd/gnd-place-v0.0.1.json 'places_gnd-gnd-place-v0.0.1-20240808'
poetry run invenio index create  -b rero_mef/concepts/gnd/mappings/v7/concepts_gnd/gnd-concept-v0.0.1.json 'concepts_gnd-gnd-concept-v0.0.1-20240808'

aliases

from invenio_search import current_search_client
current_search_client.indices.put_alias('places_gnd-gnd-place-v0.0.1-20240808', 'places_gnd')
current_search_client.indices.put_alias('places_gnd-gnd-place-v0.0.1-20240808', 'places_gnd-gnd-place-v0.0.1')
current_search_client.indices.put_alias('concepts_gnd-gnd-concept-v0.0.1-20240808', 'concepts_gnd')
current_search_client.indices.put_alias('concepts_gnd-gnd-concept-v0.0.1-20240808', 'concepts_gnd-gnd-concept-v0.0.1')

move indexes

index_name=`poetry run invenio rero es index info -i concepts_idref`
echo $index_name
poetry run invenio rero es index move concepts_idref $index_name concepts_idref-idref_concept-v1.0.0-20240808 -v
poetry run invenio index delete $index_name

index_name=`poetry run invenio rero es index info -i concepts_rero`
echo $index_name
poetry run invenio rero es index move concepts_rero $index_name concepts_rero-rero_concept-v1.0.0-20240808 -v
poetry run invenio index delete $index_name

index_name=`poetry run invenio rero es index info -i concepts_mef`
echo $index_name
poetry run invenio rero es index move concepts_mef $index_name concepts_mef-mef_concept-v1.0.0-20240808 -v
poetry run invenio index delete $index_name

index_name=`poetry run invenio rero es index info -i places_idref`
echo $index_name
poetry run invenio rero es index move places_idref $index_name places_idref-idref_place-v1.0.0-20240808 -v
poetry run invenio index delete $index_name

test mapping

poetry run invenio rero es index update-mapping

Database

poetry run invenio alembic upgrade

OAI Harvesting

poetry run invenio oaiharvester initconfig ./data/oaisources.yml -u
poetry run invenio oaiharvester harvestname -n concepts.idref -o -5 -f 1990-01-01
poetry run invenio oaiharvester harvestname -n concepts.gnd -o -5 -f 1990-01-01
poetry run invenio oaiharvester harvestname -n places.idref -o -5 -f 1990-01-01
poetry run invenio oaiharvester harvestname -n places.gnd -o -5 -f 1990-01-01

poetry run invenio oaiharvester set-last-run -n agents.idref -d 2024-10-01
poetry run invenio oaiharvester set-last-run -n agents.gnd -d 2024-10-01
poetry run invenio oaiharvester set-last-run -n concepts.idref -d 2024-10-01
poetry run invenio oaiharvester set-last-run -n concepts.gnd -d 2024-10-01
poetry run invenio oaiharvester set-last-run -n places.idref -d 2024-10-01
poetry run invenio oaiharvester set-last-run -n places.gnd -d 2024-10-01

v0.14.0 --> v0.15.0

Mapping

Elasticsearch

poetry run invenio rero es index update-mapping

Database

poetry run invenio alembic upgrade

# on `gelatine` (`eponine` test) clean WAL
ls -ltr /database/archive/
pg_archivecleanup /database/archive/`last file from archive`

v0.12.0 --> v0.14.0

create ES mapping

poetry run invenio index create  -b rero_mef/places/mef/mappings/v7/places_mef/mef-place-v0.0.1.json 'places_mef-mef-place-v0.0.1-20230823'
poetry run invenio index create  -b rero_mef/places/idref/mappings/v7/places_idref/idref-place-v0.0.1.json 'places_idref-idref-place-v0.0.1-20230823'

aliases

from invenio_search import current_search_client
current_search_client.indices.put_alias('places_mef-mef-place-v0.0.1-20230823', 'places_mef')
current_search_client.indices.put_alias('places_mef-mef-place-v0.0.1-20230823', 'places_mef-mef-place-v0.0.1')
current_search_client.indices.put_alias('places_idref-idref-place-v0.0.1-20230823', 'places_idref')
current_search_client.indices.put_alias('places_idref-idref-place-v0.0.1-20230823', 'places_idref-idref-place-v0.0.1')

test mapping

poetry run invenio rero es index update-mapping

create DB tables

poetry run invenio db create

OAIHarvesterConfig

poetry run invenio oaiharvester initconfig ./data/oaisources.yml

populate places

poetry run invenio oaiharvester harvestname -n places.idref -o -5 -f 1990-01-01

populte timelaps

poetry run invenio oaiharvester harvestname -n concepts.idref -o -5 -f 1990-01-01

create new kubernets CronJob for places.

v0.11.0 --> v0.12.0

update ES mapping

poetry run invenio rero es index update-mapping

name=concepts_idref
index_name=`poetry run invenio rero es index info -i ${name}`
echo $index_name
poetry run invenio rero es index move ${name} $index_name concepts_idref-idref-concept-v0.0.1-20230725 -v
poetry run invenio index delete $index_name

name=concepts_rero
index_name=`poetry run invenio rero es index info -i ${name}`
echo $index_name
poetry run invenio rero es index move ${name} $index_name concepts_rero-rero-concept-v0.0.1-20230725 -v
poetry run invenio index delete $index_name

name=concepts_mef
index_name=`poetry run invenio rero es index info -i ${name}`
echo $index_name
poetry run invenio rero es index move ${name} $index_name concepts_mef-mef-concept-v0.0.1-20230725 -v
poetry run invenio index delete $index_name

name=viaf
index_name=`poetry run invenio rero es index info -i ${name}`
echo $index_name
poetry run invenio rero es index move ${name} $index_name viaf-viaf-v0.0.1-20230725 -v
poetry run invenio index delete $index_name


poetry run invenio rero es index update-mapping

run alembic migration scripts :

poetry run invenio alembic upgrade