Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fuse v2 #4772

Merged
merged 70 commits into from
May 24, 2020
Merged

Fuse v2 #4772

Show file tree
Hide file tree
Changes from 54 commits
Commits
Show all changes
70 commits
Select commit Hold shift + click to select a range
0fcd089
Upgrade go-fuse.
jdoliner Mar 30, 2020
25f8f35
Adds the start of a new fuse layer.
jdoliner Mar 30, 2020
686100e
Extend fuse to be able to list files.
jdoliner Mar 30, 2020
b3d130c
Implements reading content from files.
jdoliner Mar 30, 2020
45a09b9
Reimplement commit freezing on first read.
jdoliner Mar 30, 2020
1c3dde4
Adds the start of write support.
jdoliner Mar 31, 2020
bc4f872
Keep track of which files are dirty.
jdoliner Mar 31, 2020
63ea4db
Enables writing to fuse mounts.
jdoliner Mar 31, 2020
deab496
Move new fuse implementation into the right place.
jdoliner Mar 31, 2020
7000178
Split up node code and mount code.
jdoliner Mar 31, 2020
788a8b9
Readd fuse test.
jdoliner Mar 31, 2020
d5ad6fe
Gets tests passing again.
jdoliner Apr 1, 2020
60af061
Remove write code.
jdoliner Apr 7, 2020
fb94fe8
Implement Statfs for fuse filesystems.
jdoliner Apr 8, 2020
ab553fb
Adds function to do an overlay mount.
jdoliner Apr 8, 2020
98bfb0b
Adds the start of write support.
jdoliner Apr 8, 2020
452f7bd
Adds slightly better error handling.
jdoliner Apr 9, 2020
c8f1c23
Adds the start of writes.
jdoliner Apr 13, 2020
22fb8f1
Fix appending to files that already exist.
jdoliner Apr 14, 2020
b465f8c
Fixes a few things with write mounts.
jdoliner Apr 14, 2020
b702d7f
Download the file in Lookup.
jdoliner Apr 24, 2020
9af74b9
Mirror fuse loopback files.
jdoliner Apr 24, 2020
c558923
More work on loopback implementation.
jdoliner Apr 27, 2020
7e94ca1
Loopback impl seems to mostly be working.
jdoliner Apr 27, 2020
38a1be2
Remove some annoying log messages.
jdoliner Apr 27, 2020
aa665f8
Adds file state tracking and caching.
jdoliner Apr 27, 2020
8f0bb4b
Implement writes.
jdoliner Apr 28, 2020
361d963
Make Create work.
jdoliner Apr 28, 2020
d7e92bb
Fuse2 -> Fuse
jdoliner Apr 28, 2020
d6bc0c7
Re add test file.
jdoliner Apr 28, 2020
e4a7f27
Fix issues surrounding deletes.
jdoliner Apr 29, 2020
2820c30
Adds delete to PutFileRequest.
jdoliner Apr 29, 2020
f65ee77
Implement DeleteFile in putfile streams.
jdoliner Apr 29, 2020
45b559c
Handle deletes in fuse.
jdoliner Apr 29, 2020
4b0c4c5
Make PutFileClient not choke on noops.
jdoliner Apr 29, 2020
b47bdd1
Implements os.Link for fuse mounts.
jdoliner Apr 29, 2020
cb4397d
Remove debug lines.
jdoliner Apr 29, 2020
03fecf5
Refactor into setFileState and getFileState.
jdoliner Apr 29, 2020
6d0bd95
Test suite now passes.
jdoliner Apr 30, 2020
877ab46
Adds test for Symlinking and makes it pass.
jdoliner May 1, 2020
0c6767f
Remove commented out import.
jdoliner May 1, 2020
5d98a18
Make test a bit harder by adding a directory.
jdoliner May 1, 2020
b9f03bf
Merge branch 'master' into fuse_v2
jdoliner May 1, 2020
7fec91e
Enforce write flag.
jdoliner May 4, 2020
e4e28b9
Make it possible to write to multiple repos.
jdoliner May 4, 2020
3bdc2c5
Cleanup downloaded files.
jdoliner May 4, 2020
1642508
Fix lint.
jdoliner May 4, 2020
5dfc8e8
Adds progress bars for fuse upload.
jdoliner May 5, 2020
54a6547
Extends arg parsing for repo options.
jdoliner May 6, 2020
e745312
Adds repo opts, and a test for them.
jdoliner May 11, 2020
04b2f7c
Update src/server/pfs/fuse/options.go
jdoliner May 12, 2020
2db645e
Dry off code to extract stack traces from errors.
jdoliner May 13, 2020
6c7be8e
Merge branch 'fuse_v2' of github.com:pachyderm/pachyderm into fuse_v2
jdoliner May 13, 2020
9c130f8
Make flag detection recognize wr, as well as rw.
jdoliner May 13, 2020
fa2fb72
Use errors.Is instead of direct comaprison.
jdoliner May 15, 2020
d8a7c6d
Update src/server/pfs/server/driver.go
jdoliner May 15, 2020
77f4cb1
Update src/server/pfs/server/driver.go
jdoliner May 15, 2020
c7cc2bf
Update src/server/pfs/cmds/mount_unix.go
jdoliner May 15, 2020
7e36398
Stackify the errors.
jdoliner May 15, 2020
4962284
Merge branch 'fuse_v2' of github.com:pachyderm/pachyderm into fuse_v2
jdoliner May 15, 2020
fd9fe8b
Merge branch 'master' into fuse_v2
jdoliner May 15, 2020
f8d8e8f
Stackify a few more errors.
jdoliner May 15, 2020
f096788
Merge branch 'master' into fuse_v2
jdoliner May 18, 2020
a803cac
Remove commented out code.
jdoliner May 20, 2020
83bbfd7
Merge branch 'master' into fuse_v2
jdoliner May 20, 2020
8878ff1
Make lint happy.
jdoliner May 21, 2020
c0833da
Fix inverted logic bug.
jdoliner May 21, 2020
1c2f847
Fixes TestPutFileSplit.
jdoliner May 21, 2020
cc282e5
Merge branch 'master' into fuse_v2
jdoliner May 22, 2020
eecef1f
Unmigrate progress to stackful errors.
jdoliner May 23, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
6 changes: 3 additions & 3 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ require (
github.com/gorilla/websocket v1.4.1 // indirect
github.com/grpc-ecosystem/go-grpc-middleware v1.1.0 // indirect
github.com/grpc-ecosystem/grpc-gateway v1.12.1 // indirect
github.com/hanwen/go-fuse v0.0.0-20180522155540-291273cb8ce0
github.com/hanwen/go-fuse/v2 v2.0.2
github.com/hashicorp/go-cleanhttp v0.5.1 // indirect
github.com/hashicorp/go-hclog v0.8.0 // indirect
github.com/hashicorp/go-immutable-radix v1.1.0 // indirect
Expand Down Expand Up @@ -104,9 +104,9 @@ require (
golang.org/x/net v0.0.0-20200226121028-0de0cce0169b
golang.org/x/oauth2 v0.0.0-20191202225959-858c2ad4c8b6
golang.org/x/sync v0.0.0-20190911185100-cd5d95a43a6e
golang.org/x/sys v0.0.0-20191210023423-ac6580df4449 // indirect
golang.org/x/sys v0.0.0-20191210023423-ac6580df4449
golang.org/x/time v0.0.0-20191024005414-555d28b269f0 // indirect
golang.org/x/tools v0.0.0-20200430221153-f26c0dd9827d // indirect
golang.org/x/tools v0.0.0-20200504215816-9f0e5ee6c7c4 // indirect
google.golang.org/api v0.6.0
google.golang.org/appengine v1.6.5 // indirect
google.golang.org/grpc v1.27.0
Expand Down
20 changes: 14 additions & 6 deletions go.sum
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -398,8 +398,10 @@ github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t
github.com/grpc-ecosystem/grpc-gateway v1.9.5/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
github.com/grpc-ecosystem/grpc-gateway v1.12.1 h1:zCy2xE9ablevUOrUZc3Dl72Dt+ya2FNAvC2yLYMHzi4=
github.com/grpc-ecosystem/grpc-gateway v1.12.1/go.mod h1:8XEsbTttt/W+VvjtQhLACqCisSPWTxCZ7sBRjU6iH9c=
github.com/hanwen/go-fuse v0.0.0-20180522155540-291273cb8ce0 h1:M5ITnkIvgRDjzpthOOgMS1ZlNI07hPgHahMn9pLowH4=
github.com/hanwen/go-fuse v0.0.0-20180522155540-291273cb8ce0/go.mod h1:4ZJ05v9yt5k/mcFkGvSPKJB5T8G/6nuumL63ZqlrPvI=
github.com/hanwen/go-fuse v1.0.0 h1:GxS9Zrn6c35/BnfiVsZVWmsG803xwE7eVRDvcf/BEVc=
github.com/hanwen/go-fuse v1.0.0/go.mod h1:unqXarDXqzAk0rt98O2tVndEPIpUgLD9+rwFisZH3Ok=
github.com/hanwen/go-fuse/v2 v2.0.2 h1:BtsqKI5RXOqDMnTgpCb0IWgvRgGLJdqYVZ/Hm6KgKto=
github.com/hanwen/go-fuse/v2 v2.0.2/go.mod h1:HH3ygZOoyRbP9y2q7y3+JM6hPL+Epe29IbWaS0UA81o=
github.com/hashicorp/errwrap v0.0.0-20141028054710-7554cd9344ce/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
github.com/hashicorp/errwrap v1.0.0 h1:hLrqtEDnRye3+sgx6z4qVLNuviH3MR5aQ0ykNJa/UYA=
github.com/hashicorp/errwrap v1.0.0/go.mod h1:YH+1FKiLXxHSkmPseP+kNlulaMuP3n2brvKWEqk/Jc4=
Expand Down Expand Up @@ -491,6 +493,8 @@ github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
github.com/kr/pty v1.1.5/go.mod h1:9r2w37qlBe7rQ6e1fg1S/9xpWHSnaqNdHD3WcMdbPDA=
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348 h1:MtvEpTB6LX3vkb4ax0b5D2DHbNAUsen0Gx5wZoq3lV4=
github.com/kylelemons/godebug v0.0.0-20170820004349-d65d576e9348/go.mod h1:B69LEHPfb2qLo0BaaOLcbitczOKLWTsrBG9LczfCD4k=
github.com/lib/pq v1.1.1/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/lib/pq v1.3.0 h1:/qkRGz8zljWiDcFvgpwUpwIAPu3r07TDvs3Rws+o/pU=
github.com/lib/pq v1.3.0/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
Expand Down Expand Up @@ -755,6 +759,7 @@ github.com/xlab/handysort v0.0.0-20150421192137-fb3537ed64a1/go.mod h1:QcJo0QPSf
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
github.com/xtgo/uuid v0.0.0-20140804021211-a0b114877d4c h1:3lbZUMbMiGUW/LMkfsEABsc5zNT9+b1CvsJx47JzJ8g=
github.com/xtgo/uuid v0.0.0-20140804021211-a0b114877d4c/go.mod h1:UrdRz5enIKZ63MEE3IF9l2/ebyx59GyGgPi+tICQdmM=
github.com/yuin/goldmark v1.1.25/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yuin/goldmark v1.1.27/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43 h1:+lm10QQTNSBd8DVTNGHx7o/IKu9HYDvLMffDhbyLccI=
github.com/yvasiyarov/go-metrics v0.0.0-20140926110328-57bccd1ccd43/go.mod h1:aX5oPXxHm3bOH+xeAttToC8pqch2ScQN/JoXYupl6xs=
Expand Down Expand Up @@ -925,10 +930,13 @@ golang.org/x/tools v0.0.0-20191029190741-b9c20aec41a5/go.mod h1:b+2E5dAYhXwXZwtn
golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20191130070609-6e064ea0cf2d/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200420001825-978e26b7c37c h1:JzwTM5XxGxiCwZEIZQPG46csyhWQxQlu2uSi3bEza34=
golang.org/x/tools v0.0.0-20200420001825-978e26b7c37c/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200430221153-f26c0dd9827d h1:8emZwJTManptbFnp36i5fbhO/1vUs3rfnh5e3Th8HEQ=
golang.org/x/tools v0.0.0-20200430221153-f26c0dd9827d/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/tools v0.0.0-20200130002326-2f3ba24bd6e7/go.mod h1:TB2adYChydJhpapKDTa4BR/hXlZSLoq2Wpct/0txZ28=
golang.org/x/tools v0.0.0-20200324182314-a5e5fedfe742 h1:C4Jhhgnr4nUar2rZdLD0pSC6YvxiZgmp4Fl904gxSXU=
golang.org/x/tools v0.0.0-20200324182314-a5e5fedfe742/go.mod h1:Sl4aGygMT6LrqrWclx+PTx3U+LnKx/seiNR+3G19Ar8=
golang.org/x/tools v0.0.0-20200504215816-9f0e5ee6c7c4 h1:eIjjOJUMGaX1Uop4S4spYLWurpTpmssLFW28uKQYEJc=
golang.org/x/tools v0.0.0-20200504215816-9f0e5ee6c7c4/go.mod h1:EkVYQZoAsY45+roYkvgYkIh4xh/qjgUK9TdY2XT94GE=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7 h1:9zdDQZ7Thm29KFXgAX/+yaf3eVbP7djjWp/dXAppNCc=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
Expand Down
6 changes: 5 additions & 1 deletion src/client/limit/limit.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,11 @@ func (c *concurrencyLimiter) Acquire() {
}

func (c *concurrencyLimiter) Release() {
<-c.sem
select {
case <-c.sem:
default:
panic("Release called without matching Acquire")
}
}

func (c *concurrencyLimiter) Wait() {
Expand Down
38 changes: 31 additions & 7 deletions src/client/pfs.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -945,6 +945,13 @@ type PutFileClient interface {
// recursive allows for recursive scraping of some types URLs. For example on s3:// urls.
PutFileURL(repoName string, commitID string, path string, url string, recursive bool, overwrite bool) error

// DeleteFile deletes a file from a Commit.
// DeleteFile leaves a tombstone in the Commit, assuming the file isn't written
// to later attempting to get the file from the finished commit will result in
// not found error.
// The file will of course remain intact in the Commit's parent.
DeleteFile(repoName string, commitID string, path string) error

// Close must be called after you're done using a PutFileClient.
// Further requests will throw errors.
Close() error
Expand Down Expand Up @@ -1063,6 +1070,25 @@ func (c *putFileClient) PutFileURL(repoName string, commitID string, path string
return nil
}

func (c *putFileClient) DeleteFile(repoName string, commitID string, path string) (retErr error) {
c.mu.Lock()
defer c.mu.Unlock()
if c.oneoff {
jdoliner marked this conversation as resolved.
Show resolved Hide resolved
defer func() {
if err := grpcutil.ScrubGRPC(c.Close()); err != nil && retErr == nil {
retErr = err
}
}()
}
if err := c.c.Send(&pfs.PutFileRequest{
File: NewFile(repoName, commitID, path),
Delete: true,
}); err != nil {
return grpcutil.ScrubGRPC(err)
}
return nil
}

// Close must be called after you're done using a putFileClient.
// Further requests will throw errors.
func (c *putFileClient) Close() error {
Expand Down Expand Up @@ -1405,13 +1431,11 @@ func (c APIClient) Walk(repoName string, commitID string, path string, f WalkFn)
// not found error.
// The file will of course remain intact in the Commit's parent.
func (c APIClient) DeleteFile(repoName string, commitID string, path string) error {
_, err := c.PfsAPIClient.DeleteFile(
c.Ctx(),
&pfs.DeleteFileRequest{
File: NewFile(repoName, commitID, path),
},
)
return grpcutil.ScrubGRPC(err)
pfc, err := c.newOneoffPutFileClient()
if err != nil {
return err
}
return pfc.DeleteFile(repoName, commitID, path)
}

type putFileWriteCloser struct {
Expand Down