Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement new chunk directory structure #259

Merged
merged 1 commit into from Nov 10, 2017

Conversation

@gilbertchen
Copy link
Owner

commented Nov 7, 2017

This PR implements the new chunk directory structure proposed in #222:

  • For storages initialized by earlier versions, there are no changes.
  • New storages initialized now will have a key fixed-nesting set to true in the config file.
  • This key forces all chunks, regardless of the storage type, to be saved with a nesting level of 1, so a chunk with an id abcdef... will be saved as chunks\ab\cdef....
  • The nesting level can be adjusted by adding a json file named nesting to the storage next to the config file, which should include two keys: read-levels and write-level
  • The value of read-levels is an array of integers representing which levels the chunks should be read from. For example, "read-levels": [0, 1, 2] means chunks must be searched at level 0, level 1, and level 2, in that order
  • The value of write-level is an integer representing the level that chunks should be written to. For example "write-level": 1 means new chunks will be written to level 1.
@@ -114,6 +119,9 @@ func (storage *GCDStorage) shouldRetry(threadIndex int, err error) (bool, error)
return true, nil
}

// convertFilePath converts the path for a fossil in the form of 'chunks/id.fsl' to 'fossils/id'. This is because
// ACD doesn't support file renaming. Instead, it only allows one file to be moved from one directory to another.

This comment has been minimized.

Copy link
@TheBestPessimist

TheBestPessimist Nov 8, 2017

Contributor

Here it should say GCD

@@ -328,8 +358,9 @@ func CreateGCDStorage(tokenFile string, storagePath string, threads int) (storag

storage.isConnected = true

storage.DerivedStorage = storage
storage.SetDefaultNestingLevels([]int{0}, 0)

This comment has been minimized.

Copy link
@TheBestPessimist

TheBestPessimist Nov 8, 2017

Contributor

I don't understand most of what you did here so this question may be stupid, but why here the default nesting levels is 0 and 0, instead of the "default" 1 and 1? Note that i also saw this nesting for other storages unrelated to google: s3cstorage, or azurestorage.

This comment has been minimized.

Copy link
@gilbertchen

gilbertchen Nov 9, 2017

Author Owner

This is the backward-compatible default, which could be either 2,3 and 2, or 0 and 0, depending on the storage type, so it has to be set in the constructor. The new "default" 1 and 1 is the same for all storage, so it can be set in SetNestingLevels.

This comment has been minimized.

Copy link
@TheBestPessimist

TheBestPessimist Nov 9, 2017

Contributor

In that case, it is too much to make a note of this somewhere? Let's say maybe in the class (is it a class?) StorageBase?

@gilbertchen

This comment has been minimized.

Copy link
Owner Author

commented Nov 10, 2017

Thank you for reviewing it. I'll make suggested changes in a separate checkin.

@gilbertchen gilbertchen merged commit 669d5ed into master Nov 10, 2017

1 check passed

license/cla Contributor License Agreement is signed.
Details

@gilbertchen gilbertchen deleted the fixed_nesting branch Nov 10, 2017

@TheBestPessimist

This comment has been minimized.

Copy link
Contributor

commented Nov 10, 2017

Sorry for not doing more. This is more than a little over my skills :(

@lowne

This comment has been minimized.

Copy link
Contributor

commented Nov 10, 2017

@gilbertchen is there any way to add the FixedNesting flag on existing (encrypted, and compliant) storages? 😭

@gilbertchen

This comment has been minimized.

Copy link
Owner Author

commented Nov 11, 2017

@lowne there is no straightforward way, but I think you can run add to create a new storage and then run copy to copy everything over. If it is an SFTP server you can run these two commands locally.

@lowne

This comment has been minimized.

Copy link
Contributor

commented Nov 11, 2017

@gilbertchen that wouldn't work for me for a number of reasons. However I solved via #264 🎉

@gilbertchen

This comment has been minimized.

Copy link
Owner Author

commented Sep 19, 2018

This pull request has been mentioned on Duplicacy Forum. There might be relevant details there:

http://forum.duplicacy.com/t/why-does-it-take-so-long-to-delete-a-single-revision/1275/4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.