Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix: Directory name infinite concatenation, also fixing issue #9 #11

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Fix: Directory name infinite concatenation, also fixing issue #9 #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d003793
Fix: Directory name infinite concatenation
Avielyo10 Feb 24, 2020
44d6c0c
Pulling the manifest properly
Avielyo10 Feb 25, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion main.go
View file
Open in desktop
Expand Up @@ -66,7 +66,7 @@ func restoreTable(bucket, prefix, tableName string, batchSize int64, waitPeriod
}

// Pull the manifest from s3 and load it to memory
err = store.LoadManifest(&storage.FileInput{Bucket: aws.String(bucket), Path: aws.String(fmt.Sprintf("%s/_SUCCESS", prefix))})
err = store.LoadManifest(&storage.FileInput{Bucket: aws.String(bucket), Path: aws.String(fmt.Sprintf("%s/manifest", prefix))})
if err != nil {
log.Fatalf("[ERROR] Unable to load the manifest flag information: %s\nAborting...\n", err)
}
Expand Down
9 changes: 8 additions & 1 deletion storage/s3.go
View file
Open in desktop
Expand Up @@ -8,6 +8,7 @@ import (
"io"
"log"
"net/url"
"strings"
"sync"
"time"

Expand Down Expand Up @@ -146,7 +147,13 @@ func (h *S3Backup) WriteToDB(tableName string, batchSize int64, waitPeriod time.
// DumpBuffer dumps the content of the given buffer to a new randomly generated
// file name in the given s3 path in the given bucket and resets the said buffer
func (h *S3Backup) DumpBuffer(input *FileInput, buff *bytes.Buffer) {
*input.Path = fmt.Sprintf("%s/%s", *input.Path, genNewFileName())
var splitPath = strings.Split(*input.Path, "/")
if len(splitPath) == 2 {
splitPath[1] = genNewFileName()
*input.Path = fmt.Sprintf("%s/%s", splitPath[0], splitPath[1])
} else {
*input.Path = fmt.Sprintf("%s/%s", *input.Path, genNewFileName())
}
if err := h.Flush(input, buff.Bytes()); err != nil {
log.Printf("[ERROR] while writing the file %s: %s", *input.Path, err)
}
Expand Down