Make framework robust to network errors (caching, alternate download paths, etc)

[cmungall]

Currently there is a network dependency if a schema imports another schema via URL - this is most commonly done via imports of types from linkml-model

This morning a number of us have seen intermittent issues with w3id, specifically for https://w3id.org/linkml/types.yaml. If w3id network resolution fails, then most test suites of both linkml itself and libs that use linkml will failm due to the need to import types.yaml over the network

e.g. linkml-runtime fail earlier today:

https://github.com/linkml/linkml-runtime/runs/4461013582?check_suite_focus=true#step:5:238

Reported here: perma-id/w3id.org#2434

They suggest:

I'd also consider adding a fallback to local cached resources if possible rather than depending on networked resources to always be available.

I think we should implement this

[cmungall]

Two orthogonal approaches:

1. a global configuration that includes a CURIE to location map, analogous to catalog-v001.xml in OWL
2. a caching mechanism

For 1, could be

download_map:
    "linkml:types":
       alternate_paths:             ## in priority order
         - https://raw.github./.....
         - /path/to/my/folder/types.yaml

For 2, one challenge is refreshes. Separate from this, it would be good to able to state the equivalent of version specifiers on imports, so that tooling can cache on a per-version basis. But this would require something like a repository for schemas...

[joeflack4]

Wouldn't /path/to/my/folder/types.yaml in (1) indeed be a locally cached file? That would seem to me to entail (2).

The way (1) is set up looks fine. As it's setup, I imagine that alternate_paths would have 1+ URLs at the top of the list, and 1 (would you ever need more than 1?) path to a locally cached file.

I don't see any harm in making this structure more explicit, as well as adding versioning, e.g.:

download_map:
    "linkml:types":
        latest:
            online:
                - https://raw.github.somerepo1/.....
                - https://raw.github.somerepo2/.....
            cache:
                - /path/to/my/folder/types_yaml/latest/types.yaml
        1.0.1:
            online:
                - https://raw.github.somerepo1/.....
                - https://raw.github.somerepo2/.....
            cache:
                - /path/to/my/folder/types_yaml/1.0.1/types.yaml

I think this is pretty self explanatory, but basically, we store this structure. Then linkml attempts to fetch from 'online' for the given version, else loads from cache.

[cmungall]

So 1 could potentially be a a map from a schema CURIE to an alternate URL, such as a raw github URL, or another server

I like the proposed datastructure, but it brings up the question of version metadata. For any given schema, including the linkml-model itself, where does info about versions live? Currently this is github. This works fine, but it might be useful to have a more explicit registry of versions, perhaps as a part of https://github.com/linkml/linkml-registry

[joeflack4]

Didn't even know that existed! I have no objections to a / that registry.

[sierra-moxon]

running into this one again this morning during github's unicorn take over.

[cmungall]

It seems there are actually two network dependencies: - imports for an external, e.g. linkml:types - prefixcommons (even importing curie_util invokes a network call, regardless of whether the model includes a default_curi_prefixes or not)

…

On Tue, Feb 22, 2022 at 8:49 AM Sierra Moxon ***@***.***> wrote: running into this one again this morning during github's unicorn take over. — Reply to this email directly, view it on GitHub <#502 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAMMOK6QRPPT3HQLVIABADU4O5CFANCNFSM5JUNOHHA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you authored the thread.Message ID: ***@***.***>