Import DVC remote plugin config schemas from plugin packages

[sisp]

Disclaimer: I'm still learning how DVC's plugin system for DVC remotes works, so please correct me if my understanding is simply wrong.

I'm working on a DVC plugin to natively support GitLab's generic package registry as a DVC remote. While doing so, I noticed that DVC manages config schemas centrally in dvc/config_schema.py for all plugins, each of which is implemented in a separate project though. This means, it's impossible to create and use a custom DVC plugin because at least the config validation fails as DVC doesn't recognize the custom plugin. The error I'm seeing when running the partially working test suite is:

dvc.config.ConfigError: config file error: Unsupported URL type gitlab:// for dictionary value @ data['remote']['upstream']

I don't think a central config schema aligns well with a plugin system. Instead, I believe it would be much better to retrieve the config schemas for the plugins from the (installed and registered) plugin packages, i.e. each plugin should define its own config schema and DVC should combine them at runtime. This approach would make DVC truly extensible - which is the whole idea of a plugin system in the first place (isn't it?).

[efiop]

It makes total sense and it is and was in our plans, but we didn't get to it since there wasn't any demand from the outside.

[skshetry]

@sisp, are you still planning on creating a dvc-plugin/fsspec?

[sisp]

Not at the moment. It turned out that GitLab's packages API doesn't properly support listing files (which would be necessary for performing garbage collection) and there's a race condition when performing parallel upload of files to a new package which causes an HTTP 500 response, which is annoying behavior. Since DVC (via scmrepo) now supports Git LFS download, this seems like the way forward for me for now.

[skshetry]

Okay then. I am closing this issue, as we don't have plans to implement this in short to medium term.

[adamlabadorf]

Reopening request — new use case: dvc-osf (Open Science Framework plugin)

Hi! I'm building dvc-osf, a DVC plugin for Open Science Framework (https://osf.io/), and I've hit exactly the same wall described in this issue — Unsupported URL type osf:// because the config schema in dvc/config_schema.py is hardcoded.

The plugin works perfectly at the filesystem level (fsspec registration, known_implementations, all unit/integration tests pass), but DVC's ByUrl validator in config_schema.py rejects the URL before the filesystem ever gets a chance to load.

I'd like to propose a concrete implementation and submit a PR. The change would be minimal and backward-compatible:

1. Filesystem plugins define a REMOTE_CONFIG class attribute with their accepted config keys
2. DVC discovers installed plugins via the existing dvc.fs entry point group at startup
3. Any discovered plugin with REMOTE_CONFIG gets merged into REMOTE_SCHEMAS and the ByUrl validator is rebuilt

Something like:

# In the plugin's filesystem class
class OSFFileSystem(ObjectFileSystem):
    protocol = "osf"
    REMOTE_CONFIG = {
        "token": str,
        "endpoint_url": str,
        "project_id": str,
        "provider": str,
    }

# In dvc/config_schema.py — called once at import time
from importlib.metadata import entry_points

def _discover_plugin_schemas():
    for ep in entry_points(group="dvc.fs"):
        try:
            cls = ep.load()
            if hasattr(cls, "REMOTE_CONFIG"):
                scheme = getattr(cls, "protocol", ep.name)
                if scheme not in REMOTE_SCHEMAS:
                    REMOTE_SCHEMAS[scheme] = {**cls.REMOTE_CONFIG, **REMOTE_COMMON}
        except Exception:
            pass
    SCHEMA["remote"] = {str: ByUrl(REMOTE_SCHEMAS)}

This keeps all existing hardcoded schemes working, adds zero overhead for users without third-party plugins, and makes DVC genuinely extensible for new storage backends.

Happy to submit the PR if you're open to it. Would appreciate guidance on preferred approach or any concerns.