[performance] update storage backend and make use of seek syscall when available #2924
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
NyaaaWhatsUpDoc
commented
May 21, 2024
| @@ -80,10 +80,6 @@ func setupPrune(ctx context.Context) (*prune, error) { | |||
| func (p *prune) shutdown() error { | |||
| errs := gtserror.NewMultiError(2) | |||
|
|
|||
| if err := p.storage.Close(); err != nil { | |||
There was a problem hiding this comment.
the new storage type doesn't bother with any kind of close mechanism as it's not really needed
…uld only be S3 now)
NyaaaWhatsUpDoc
changed the title
tsmethurst
reviewed
May 22, 2024
| if _, err := fastcopy.CopyN(io.Discard, src, start); err != nil { | ||
| log.Errorf(r.Context(), "error reading from source: %v", err) | ||
| return | ||
| if rs, ok := src.(io.ReadSeeker); ok { |
|
Good one! |
NyaaaWhatsUpDoc
added a commit
to NyaaaWhatsUpDoc/gotosocial
that referenced
this pull request
Jun 7, 2024
…n available (superseriousbusiness#2924) * update to use go-storage/ instead of go-store/v2/storage/ * pull in latest version from codeberg * remove test output 😇 * add code comments * set the exclusive bit when creating new files in disk config * bump to actual release version * bump to v0.1.1 (tis a simple no-logic change) * update readme * only use a temporary read seeker when decoding video if required (should only be S3 now) * use fastcopy library to use memory pooled buffers when calling TempFileSeeker() * update to use seek call in serveFileRange()
nyarla
pushed a commit
to nyarla/gotosocial-modded
that referenced
this pull request
Jun 19, 2024
…n available (superseriousbusiness#2924) * update to use go-storage/ instead of go-store/v2/storage/ * pull in latest version from codeberg * remove test output 😇 * add code comments * set the exclusive bit when creating new files in disk config * bump to actual release version * bump to v0.1.1 (tis a simple no-logic change) * update readme * only use a temporary read seeker when decoding video if required (should only be S3 now) * use fastcopy library to use memory pooled buffers when calling TempFileSeeker() * update to use seek call in serveFileRange()
nyarla
pushed a commit
to nyarla/gotosocial-modded
that referenced
this pull request
Jun 19, 2024
…n available (superseriousbusiness#2924) * update to use go-storage/ instead of go-store/v2/storage/ * pull in latest version from codeberg * remove test output 😇 * add code comments * set the exclusive bit when creating new files in disk config * bump to actual release version * bump to v0.1.1 (tis a simple no-logic change) * update readme * only use a temporary read seeker when decoding video if required (should only be S3 now) * use fastcopy library to use memory pooled buffers when calling TempFileSeeker() * update to use seek call in serveFileRange()
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
This moves us from using
codeberg.org/gruf/go-store/v2/storageusingcodeberg.org/gruf/go-storageas our underlying storage library. Which itself is just moving the library out of the nested repo it was in, since it expanded in usage scope from its original use. This in turn trims some of the unused dependencies from the vendor/ folder, and makes it easier updating thego-storagelibrary myself since i don't need to update the rest of the key-value store around it. (also the key-value store itself isn't really getting used anymore by any of my projects, so it may end up falling by the wayside, and that's fine)In updating to this version of the underlying storage library, since there's no more wrapping of the returned reader types, we can now rely on the seek syscall in a number of places which should speed-up file serving and video decoding on those using disk based storage :)
Checklist
go fmt ./...andgolangci-lint run.