Skip to content

Migration workflow

Jump to bottom
Pascal Repond edited this page Feb 6, 2026 · 67 revisions

Migration workflow steps

Restore test

Index:

sonar

poetry run invenio rero es snapshot create backup -n migration-20250924 -w

sonar dev

uv run invenio rero es snapshot repository create backup /invenio/storage/prod/es-backup

# clean indices
uv run invenio rero es snapshot list backup
uv run invenio rero es snapshot restore backup migration-20250924 --wait

Files [sonar-setup-shell]

apt update
apt install rsync

rsync -avr --delete --progress /invenio/storage/files_prod/* /invenio/storage/files/new_files/  --exclude=lost+found --exclude=sonar_prod

DB kerosen

export kerosen

ssh root@kerosen
su - postgres
pg_dump --clean sonar > /database/dump/sonar_prod_20251014.sql
exit

#### import sonar dev
psql -U sonar sonar < /invenio/storage/backup/dump/sonar_prod_20251014.sql
exit

Correct problems

poetry run /invenio/storage/scripts/correct_sonar.py utils correct-wrong-pids -t doc -v -c

v1.12.3 --> v1.12.4

docker

  • Change grobid version to grobid/grobid:0.8.2-crf

projects data

  • for all projects where organisation is not "hepvs", delete validation field
  • for all hepvs projects where there is a field validation.logs:
    • for each log entry where user contains a "name" and "pid", change it to user.$ref with the user as a dict containing $ref, first_name, last_name and pid.
"""Migrate project validation data.

1. Remove validation field from non-hepvs projects.
2. Migrate validation.logs[].user entries from the old format:
       {"name": "Sophie Roh", "pid": "18"}
   to the new format:
       {"$ref": "https://sonar.ch/api/users/18", "pid": "18", "first_name": "Sophie", "last_name": "Roh"}
"""

import copy

from invenio_db import db
from sqlalchemy.orm.attributes import flag_modified

from sonar.modules.users.api import UserRecord
from sonar.resources.projects.models import RecordMetadata

HEPVS_ORG_SUFFIX = "/organisations/hepvs"

cleaned = 0
migrated = 0
skipped = 0
errors = 0

for record in RecordMetadata.query.all():
    data = record.json
    metadata = data.get("metadata", {})

    if "validation" not in metadata:
        continue

    org_ref = metadata.get("organisation", {}).get("$ref", "")

    # Step 1: remove validation from non-hepvs projects
    if not org_ref.endswith(HEPVS_ORG_SUFFIX):
        del metadata["validation"]
        flag_modified(record, "json")
        cleaned += 1
        print(f"  [CLEAN] Record {record.id}: validation removed (org: {org_ref})")
        continue

    # Step 2: migrate validation logs for hepvs projects
    logs = metadata["validation"].get("logs", [])
    if not logs:
        continue

    logs_copy = copy.deepcopy(logs)
    changed = False

    try:
        for log in logs_copy:
            user = log.get("user", {})

            # Already migrated
            if "$ref" in user and "first_name" in user:
                continue

            pid = user.get("pid")
            if not pid:
                print(f"  [SKIP] Record {record.id}: log user has no pid: {user}")
                skipped += 1
                continue

            # Build $ref if missing
            if "$ref" not in user:
                user["$ref"] = f"https://sonar.ch/api/users/{pid}"

            # Resolve first_name/last_name from user record
            if "first_name" not in user or "last_name" not in user:
                try:
                    user_record = UserRecord.get_record_by_pid(pid)
                except Exception:
                    user_record = None

                if user_record:
                    user["first_name"] = user_record.get("first_name", "")
                    user["last_name"] = user_record.get("last_name", "")
                else:
                    # Fallback: split "name" field if present
                    name = user.pop("name", "")
                    parts = name.rsplit(" ", 1) if name else ["", ""]
                    user["first_name"] = parts[0] if len(parts) > 1 else ""
                    user["last_name"] = parts[-1]
                    print(f"  [WARN] Record {record.id}: user pid={pid} not found, used name fallback: {name}")

            # Remove old "name" field if present
            user.pop("name", None)

            log["user"] = {
                "$ref": user["$ref"],
                "pid": user["pid"],
                "first_name": user["first_name"],
                "last_name": user["last_name"],
            }
            changed = True

        if changed:
            metadata["validation"]["logs"] = logs_copy
            flag_modified(record, "json")
            migrated += 1
            print(f"  [OK] Record {record.id}: {len(logs_copy)} log(s) migrated")

    except Exception as exc:
        errors += 1
        print(f"  [ERROR] Record {record.id}: {exc}")

db.session.commit()
print(f"\nDone: {cleaned} cleaned, {migrated} migrated, {skipped} skipped, {errors} errors.")

v1.11.0 --> v1.12.0

Alembic

uv run invenio alembic stamp 428b919be0ea
uv run invenio alembic upgrade

Update mapping

from invenio_search import current_search_client
current_search_client.indices.create('records-record-v1.0.0')
uv run invenio rero es index update-mapping

Documents

Remove _oai in documents.

# estimation for 300'000 records: 5h
from invenio_pidstore.models import PersistentIdentifier
from rero_invenio_base.modules.tasks import run_on_worker
from rero_invenio_base.modules.utils import chunk


code = '''
def pop_oai(_ids):
    from sonar.modules.documents.api import DocumentRecord
    from invenio_db import db
    success = 0
    failed = 0
    for uuid in _ids:
        try:
            record = DocumentRecord.get_record(uuid)
            if record.get('_oai'):
                record['_oai'].pop('updated', None)
                record['_oai'].pop('sets', None)
                record.commit()
                db.session.commit()
                record.reindex()
                success += 1
        except Exception as err:
            print(f"Exception {uuid}: {err}")
            failed += 1
    return {"success": success, "failed": failed}
'''


parallel = 7
ids = [pid.object_uuid for pid in PersistentIdentifier.query.filter_by(pid_type='doc').filter_by(status='R').all()]
for count, c in enumerate(chunk([str(val) for val in ids], len(ids) // parallel), 1):
    res = run_on_worker.delay(code, 'pop_oai', c)
    print('documents', count, len(c), res)
index_name=`uv run invenio rero es index info -i documents |grep -v percolator`
echo $index_name
uv run invenio rero es index move documents $index_name documents-document-v1.0.0-20250923 -v
uv run invenio index delete $index_name --yes-i-know

Deposits

Remove category from deposit.

from sonar.modules.deposits.api import DepositRecord
n = 0
for pid in PersistentIdentifier.query.filter_by(
            pid_type='depo').filter_by(status='R').all():
    try:
        record = DepositRecord.get_record(pid.object_uuid) 
        n += 1
        changed = False
        for f in record.get('_files', []):
            if 'category' in f:
                del f['category']
                changed = True
            if 'embargo' in f:
                del f['embargo']
                changed = True
        if changed:
            record.commit()
            db.session.commit()
            record.reindex()
    except:
        print(f"Failed to update record {pid.pid_value}")

print(f"Updated {n} records")
index_name=`uv run invenio rero es index info -i deposits | grep -v percolator`
echo $index_name
uv run invenio rero es index move deposits $index_name deposits-deposit-v1.0.0-20250923 -v
uv run invenio index delete $index_name --yes-i-know

Organisations

index_name=`uv run invenio rero es index info -i organisations | grep -v percolator`
echo $index_name
uv run invenio rero es index move organisations $index_name organisations-organisation-v1.0.0-20250923 -v
uv run invenio index delete $index_name --yes-i-know

Collections

index_name=`uv run invenio rero es index info -i collections | grep -v percolator`
echo $index_name
uv run invenio rero es index move collections $index_name collections-collection-v1.0.0-20250923 -v
uv run invenio index delete $index_name --yes-i-know

ConfigMap

  • replace INVENIO_SQLALCHEMY_POOL_RECYCLE by INVENIO_SQLALCHEMY_ENGINE_OPTIONS: '{ "pool_recycle": 360 }'
  • set the config in the sqlalchemy container/server by setting PGOPTIONS to -c statement_timeout=30s -c idle_in_transaction_session_timeout=60s
  • add IDP_CERTIFICATES_DIR: /invenio/storage/prod/data/idp_certificates
  • add SONAR_FILES_PATH: /invenio/storage/files
  • add PYTHONWARNINGS: ignore

Post migration

Clone this wiki locally