Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Import/export CAR DAG archives #1798

Open
Tracked by #109
lidel opened this issue Jun 1, 2021 · 7 comments · Fixed by #1837
Open
Tracked by #109

Import/export CAR DAG archives #1798

lidel opened this issue Jun 1, 2021 · 7 comments · Fixed by #1837
Labels
area/screen/files Issues related to Files screen kind/enhancement A net-new feature or improvement to an existing feature P1 High: Likely tackled by core team if no one steps up

Comments

@lidel
Copy link
Member

lidel commented Jun 1, 2021

Given that:

  • ipfs dag import|export exists
  • go-ipfs 0.9+ supports exporting DAG archive via any public gateway (/api/v0/dag/export)
  • tools like ipfs-car and ipfs-content-addressed-archiver make it easier for people to produce/consume content-addressed archives

We should support "DAG import/export" in ipfs-webui somehow.

Import DAG

Import could work the same as "From IPFS" :

2021-06-01--13-01-05

We most likely want to add an explicit "From CAR Archive" item here (personally, I like the idea of a whimsical car icon :)), but "stroke data" from ipfs-css would also do.

Export DAG

Add complimentary "Export DAG" to context menu on Files screen:

2021-06-11--16-08-29

@lidel lidel added area/screen/files Issues related to Files screen need/triage Needs initial labeling and prioritization labels Jun 1, 2021
@lidel lidel changed the title Import DAG archives Import/export DAG archives Jun 11, 2021
@lidel lidel added kind/enhancement A net-new feature or improvement to an existing feature P1 High: Likely tackled by core team if no one steps up and removed need/triage Needs initial labeling and prioritization labels Jun 11, 2021
@lidel lidel assigned olizilla and unassigned olizilla Jun 18, 2021
@Gozala
Copy link
Contributor

Gozala commented Jun 23, 2021

Seems like it would be a good idea to do it after ipfs/js-ipfs#2745

@lidel lidel changed the title Import/export DAG archives Import/export CAR DAG archives Aug 17, 2021
@hacdias hacdias self-assigned this Aug 18, 2021
@hacdias hacdias linked a pull request Aug 18, 2021 that will close this issue
@hacdias hacdias linked a pull request Aug 18, 2021 that will close this issue
@hacdias
Copy link
Member

hacdias commented Sep 1, 2021

@lidel as I mentioned on #1838 (comment), we'll need to update to the latest HTTP client across WebUI and the Explore page to support the import part. The export is done on #1837. I don't think I should work on the upgrade right now as that would take a long time working only 10h a week. Updating the http client will also require updating everything else connected: multiformats, ipld, multicodecs, etc.

I think we could merge the export button and evaluate how much effort is needed for the import part.

@lidel
Copy link
Member Author

lidel commented Sep 2, 2021

On ipfs-http-client API update

Agree that upgrading http-client and all the deps (especially ipld-explorer-components) will be a time sink (due to API changes and refactors).
I'll block some time on Friday to understand the scope bit better and decide what's next.

On import UX

In regard to CAR Import/export – I am a bit hesitant to introduce export without the ability to import it as well.
Main worry: people will import .car via Import→File and produce garbage blocks (CAR with original DAG wrapped into redundant dag-pb DAG)

I think we not only need Import→DAG Archive but also make Import→File check for magic bytes and do DAG import instead of importing CAR as binary unixfs file.

@hacdias
Copy link
Member

hacdias commented Sep 2, 2021

@lidel I agree with your worry. I didn't think about that, but it's a good point.

@lidel
Copy link
Member Author

lidel commented Jun 9, 2022

Reopening since we also need #1838

@lidel lidel reopened this Jun 9, 2022
@hsn10
Copy link

hsn10 commented Jul 5, 2022

Add maximum CAR size so output can be split into multiple files. Standard between storage services seems to be accept 100 MB CAR files for uploads.

@lidel
Copy link
Member Author

lidel commented Jul 6, 2022

@hsn10 that is an artificial limitation of some specific services. Users of such services can do the splitting with third-party tools like carbites.

We could integrate carbites splitter into ipfs-webui, but it would be pretty expensive to implement, and require additional UI design (how to opt into splitting, how to control splitting parameters, what should be the default).
Definitely not in the scope here, but if someone proposes UX design for this in a NEW issue, we can discuss feasibility there 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/screen/files Issues related to Files screen kind/enhancement A net-new feature or improvement to an existing feature P1 High: Likely tackled by core team if no one steps up
Projects
No open projects
Archived in project
Status: Needs Grooming
Development

Successfully merging a pull request may close this issue.

5 participants