import: from external git does not import lfs-tracked file content

[morteza]

Bug Report

Description

I encountered an issue when trying to import an external git repository that contains files tracked by git-lfs (using either import or get). Files not tracked by git-lfs are imported correctly, however the lfs-tracked files only contain pointers to the lfs instead of the actual content.

Reproduce

1. Run dvc import https://huggingface.co/datasets/morteza/cogtext.git gpt3/abstracts_gpt3ada.nc
2. Run dvc pull
3. Open the abstracts_gpt3ada.nc file, which should contain the actual binary content but instead contains text pointers to the lfs object.

Expected

When importing LFS-tracked files from an external git repository, the files should contain the actual content and not text pointers.

Environment information

DVC version: 2.47.2 (pip)
-------------------------
Platform: Python 3.11.0 on Linux-5.15.0-67-generic-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 0.42.3
        dvc_objects = 0.21.1
        dvc_render = 0.2.0
        dvc_task = 0.2.0
        scmrepo = 0.1.15
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        ssh (sshfs = 2023.1.0)
Cache types: hardlink, symlink
Cache directory: ext4 on /dev/sdb5
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/sdb5
Repo: dvc, git

[pmrowla]

LFS is a 3rd party git extension and is not currently supported in our git backend. As you noted, on import we will only end up tracking what git itself returns (without the use of the additional LFS extension), which is just the LFS text file.

[dberenbaum]

Reopening since it's a legitimate feature request and the most concrete request we have for how to work with huggingface datasets in dvc.

[Taytay]

@dberenbaum : It's exactly this HuggingFace integration that bit me! Thank you for considering it.

[efiop]

For the record: https://github.com/liberapay/git-lfs-fetch.py/blob/master/git_lfs/__init__.py might be useful to look at. The general feeling is that we will have to either embed it into scmrepo's gitfs or create a new fs implementation.

[pmrowla]

The LFS spec is well defined (https://github.com/git-lfs/git-lfs/blob/main/docs/spec.md), we don't actually need a separate fs implementation. scmrepo just needs to support the proper LFS git smudge and clean filters, and then the existing GitFileSystem would work out of the box (since when we load the LFS objects it would run the filter and fetch/load the actual LFS file and not the plaintext placeholder.

(If we add the requirement that users have the LFS client installed, we just need the general filter support and can call the existing LFS commands the same way we call credential helpers. But the actual LFS client behavior is simple enough that we could implement our own client so that DVC/scmrepo could read from LFS repos without the user installing LFS)