-
Notifications
You must be signed in to change notification settings - Fork 1.3k
Description
Problem
Third-party DVC filesystem plugins (installed via the dvc.fs entry point group) cannot be used as DVC remotes without modifying DVC core. The config validator in dvc/config_schema.py maintains a hardcoded list of URL schemes in REMOTE_SCHEMAS, so any plugin with a custom scheme gets rejected immediately:
dvc.config.ConfigError: config file error: Unsupported URL type osf:// for dictionary value @ data['remote']['myremote']
This was previously raised in #9711, which was closed due to lack of demand. I'm reopening as a concrete proposal with a working implementation.
Proposed Solution
Allow plugins to declare a REMOTE_CONFIG class attribute (a dict of config keys → validators) on their filesystem class. At import time, _discover_plugin_schemas() iterates over installed dvc.fs entry points and merges any declared schemas into REMOTE_SCHEMAS.
Plugin side (no DVC core changes needed by the plugin author beyond declaring the attribute):
class OSFFileSystem(ObjectFileSystem):
protocol = "osf"
REMOTE_CONFIG: ClassVar[dict] = {
"token": str,
"project_id": str,
"provider": str,
"endpoint_url": str,
}DVC core change (~25 lines in dvc/config_schema.py):
from importlib.metadata import entry_points
def _discover_plugin_schemas():
for ep in entry_points(group="dvc.fs"):
try:
cls = ep.load()
except Exception: # noqa: BLE001,S112
continue
remote_config = getattr(cls, "REMOTE_CONFIG", None)
if not remote_config:
continue
protocol = getattr(cls, "protocol", ep.name)
schemes = (protocol,) if isinstance(protocol, str) else protocol
for scheme in schemes:
if scheme not in REMOTE_SCHEMAS:
REMOTE_SCHEMAS[scheme] = {**remote_config, **REMOTE_COMMON}
_discover_plugin_schemas()Properties
- ✅ Backward compatible — existing hardcoded schemes are never overwritten
- ✅ Zero overhead for users without third-party plugins (entry point group is empty)
- ✅ Graceful degradation — plugins that fail to load are silently skipped
- ✅ Minimal change — ~25 lines added to
config_schema.py, no other files touched - ✅ Tested — 7 new unit tests covering schema registration, skipping, multi-protocol, load failures, and ByUrl validation
Working Implementation
I have a working implementation in adamlabadorf/dvc@feature/plugin-schema-discovery, developed alongside dvc-osf, a plugin for Open Science Framework storage.
The dvc-osf integration tests (30 passing) confirm that with this change, dvc remote add, dvc push, and dvc pull all work correctly with a third-party plugin.
Happy to submit a PR — would appreciate any guidance on approach or concerns before doing so.