forked from fsouza/fake-gcs-server
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Store object data natively and metadata in xattr
The latter remains a JSON-encoded blob but is now stored in file extended attributes, except on Windows where it is a separate file. This reduces memory usage and is much faster by avoiding JSON-encoding large objects. This enables a future commit to avoid reading the entire object, particularly for range requests. Note that this commit changes the on-disk format and is not compatible with previous data sets. Extended attributes have some caveats including lack of tmpfs and Windows support. References pkg/xattr#47. References fsouza#669. Fixes fsouza#671.
- Loading branch information
Showing
9 changed files
with
115 additions
and
15 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| package backend | ||
|
|
||
| import ( | ||
| "github.com/pkg/xattr" | ||
| ) | ||
|
|
||
| const XATTR_KEY = "user.metadata" | ||
|
|
||
| func writeXattr(path string, encoded []byte) error { | ||
| return xattr.Set(path, XATTR_KEY, encoded) | ||
| } | ||
|
|
||
| func readXattr(path string) ([]byte, error) { | ||
| return xattr.Get(path, XATTR_KEY) | ||
| } | ||
|
|
||
| func isXattrFile(path string) bool { | ||
| return false | ||
| } | ||
|
|
||
| func removeXattrFile(path string) error { | ||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,23 @@ | ||
| package backend | ||
|
|
||
| import ( | ||
| "github.com/pkg/xattr" | ||
| ) | ||
|
|
||
| const XATTR_KEY = "user.metadata" | ||
|
|
||
| func writeXattr(path string, encoded []byte) error { | ||
| return xattr.Set(path, XATTR_KEY, encoded) | ||
| } | ||
|
|
||
| func readXattr(path string) ([]byte, error) { | ||
| return xattr.Get(path, XATTR_KEY) | ||
| } | ||
|
|
||
| func isXattrFile(path string) bool { | ||
| return false | ||
| } | ||
|
|
||
| func removeXattrFile(path string) error { | ||
| return nil | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| package backend | ||
|
|
||
| import ( | ||
| "io/ioutil" | ||
| "os" | ||
| "strings" | ||
| ) | ||
|
|
||
| const XATTR_SUFFIX = ".metadata" | ||
|
|
||
| func writeXattr(path string, encoded []byte) error { | ||
| return ioutil.WriteFile(path+XATTR_SUFFIX, encoded, 0o600) | ||
| } | ||
|
|
||
| func readXattr(path string) ([]byte, error) { | ||
| return ioutil.ReadFile(path + XATTR_SUFFIX) | ||
| } | ||
|
|
||
| func isXattrFile(path string) bool { | ||
| return strings.HasSuffix(path, XATTR_SUFFIX) | ||
| } | ||
|
|
||
| func removeXattrFile(path string) error { | ||
| return os.Remove(path + XATTR_SUFFIX) | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters