Wrap lobjs so they can be deduped w/ other IPFS objects, fixes #12 #21

sameer · 2019-01-04T15:12:00Z

This PR fixes #12 by splitting large objects into a header and raw data which are individually hashed then patched together for the lobj hash. I kept a fallback to just adding the entire object if the header can't be found.

I ran it against a repo, one 3.6MB lobj example is:

headerHash = QmddzD2eZgPcLHYku5DMFXgBXh8GPzPGr3SaBr4ioPuVDU
dataHash= QmTBDL8wuSbjF5eCzkwdPXA2sTiTNmwKtqUUkJTYWKWa4W
objectHash = QmURkbZP6hVdt4MAo9CBv4rJxNUA7x57UKgHtzxgpfY8Np

Not exactly sure how to add tests for this -- any guidance is appreciated.

Edit: tests seem to fail on macOS but I think it's unrelated.

magik6k

1 small thing. A test or two would be nice too.

magik6k · 2019-01-07T12:30:30Z

cmd/git-remote-ipld/ipld.go

+					return err
+				}
+
+				dataHash, err := h.api.Add(bytes.NewReader(data[endOfHeader:]))


These two adds can run in parallel with sync.waitgroup (api calls can be slow, but can be heavily parallelized)

sameer · 2019-01-07T18:42:48Z

So I added a lobj branch to the mock git repo with a file large enough to generate a lobj that would hit my changes on a push. However, IpnsHandler.Push() is not called because the testing never does an actual push. I will try to add a push test.

Edit: I misspoke, there is a push test, but the large object does not get sent because it's pushing refs/heads/master.

sameer · 2019-01-07T21:55:07Z

Discovered that the push tests were never actually run because there was a missing newline at the end of testCase(t, args, "push refs/heads/master:refs/heads/master\n", []string{}).

magik6k · 2019-01-08T01:51:51Z

Jenkins is broken in ipfs-shipyard for go projects, I'll try to work around it tomorrow if time permits

sameer · 2019-01-23T21:07:19Z

Looks like macOS is still failing:

[git-remote-ipld] Running shell script

+ go test -v ./...

+ tee output

/var/jenkins/workspace/pyard_git-remote-ipld_PR-21-BEUY3MIKEUQKTXCLQUO4RZM773OQDB3XCGN6CAIWBSB6ADFZOXKQ/src/github.com/ipfs shipyard/git-remote-ipld/cmd/git-remote-ipld/ipld.go:15:2: cannot find package "github.com/ipfs-shipyard/git-remote-ipld/core" in any of:

	/var/jenkins/tools/org.jenkinsci.plugins.golang.GolangInstallation/1.11/src/github.com/ipfs-shipyard/git-remote-ipld/core (from $GOROOT)

	/var/jenkins/workspace/pyard_git-remote-ipld_PR-21-BEUY3MIKEUQKTXCLQUO4RZM773OQDB3XCGN6CAIWBSB6ADFZOXKQ/src/github.com/ipfs-shipyard/git-remote-ipld/core (from $GOPATH)

…hipyard#12

magik6k self-requested a review January 4, 2019 15:30

magik6k reviewed Jan 7, 2019

View reviewed changes

sameer force-pushed the lobj-wrap branch 2 times, most recently from 8b94056 to ca3d565 Compare January 7, 2019 18:37

sameer force-pushed the lobj-wrap branch from ca3d565 to ac46ef4 Compare January 7, 2019 22:07

magik6k self-requested a review January 8, 2019 01:51

sameer force-pushed the lobj-wrap branch from ac46ef4 to d03c656 Compare January 23, 2019 21:03

Wrap lobjs so they can be deduped w/ other IPFS objects, fixes ipfs-s…

71d1d0a

…hipyard#12

sameer force-pushed the lobj-wrap branch from d03c656 to 71d1d0a Compare March 13, 2019 19:32

sameer closed this Aug 3, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Wrap lobjs so they can be deduped w/ other IPFS objects, fixes #12 #21

Wrap lobjs so they can be deduped w/ other IPFS objects, fixes #12 #21

sameer commented Jan 4, 2019 •

edited

magik6k left a comment

magik6k Jan 7, 2019

sameer commented Jan 7, 2019 •

edited

sameer commented Jan 7, 2019 •

edited

magik6k commented Jan 8, 2019 •

edited

sameer commented Jan 23, 2019

Wrap lobjs so they can be deduped w/ other IPFS objects, fixes #12 #21

Wrap lobjs so they can be deduped w/ other IPFS objects, fixes #12 #21

Conversation

sameer commented Jan 4, 2019 • edited

magik6k left a comment

Choose a reason for hiding this comment

magik6k Jan 7, 2019

Choose a reason for hiding this comment

sameer commented Jan 7, 2019 • edited

sameer commented Jan 7, 2019 • edited

magik6k commented Jan 8, 2019 • edited

sameer commented Jan 23, 2019

sameer commented Jan 4, 2019 •

edited

sameer commented Jan 7, 2019 •

edited

sameer commented Jan 7, 2019 •

edited

magik6k commented Jan 8, 2019 •

edited