Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use file name as archive description #6

Closed
Jonny007-MKD opened this issue Mar 11, 2017 · 27 comments
Closed

Use file name as archive description #6

Jonny007-MKD opened this issue Mar 11, 2017 · 27 comments

Comments

@Jonny007-MKD
Copy link
Contributor

I think it would be nice to use the file name (date, time, hash/rand, e.g. 20170309-140558-78a43) as archive description for all subordinated files.

These two files belong together and could be grouped by the description:

... |                         Archive Description                         |       Uploaded       | SHA256 tree hash | ...
... | /temp/iceshelf/20170309-140558-78a43.tar.gpg.sig.vol01+2.par2.sig   | 2017-03-09T14:21:32Z | 294e1fca1...     | ...
... | /temp/iceshelf/20170309-140558-78a43.tar.gpg.sig.vol03+4.par2.sig   | 2017-03-09T14:21:48Z | 6dd92125e...     | ...
@mrworf
Copy link
Owner

mrworf commented Mar 12, 2017

I'm not sure I understand, it will re-use the same vault, and all the files generated are prefixed with the unique id of the job. Latest version of iceshelf fixed that (previously, filelist.txt was not unique, but now it is)

@Jonny007-MKD
Copy link
Contributor Author

Jonny007-MKD commented Mar 12, 2017 via email

@mrworf
Copy link
Owner

mrworf commented Mar 12, 2017

Ahh, oki :)

I'll see what I can do. Should be able to get that in today.

@Jonny007-MKD
Copy link
Contributor Author

Jonny007-MKD commented Mar 12, 2017 via email

@mrworf
Copy link
Owner

mrworf commented Mar 12, 2017

Please try the latest version, it should use description AND remove the path from the filenames.

@Jonny007-MKD
Copy link
Contributor Author

Jonny007-MKD commented Mar 13, 2017 via email

@mrworf
Copy link
Owner

mrworf commented Mar 13, 2017 via email

@Jonny007-MKD
Copy link
Contributor Author

Jonny007-MKD commented Mar 13, 2017 via email

@mrworf
Copy link
Owner

mrworf commented Mar 14, 2017

You're right, only works once 😞 so for now it's gone and it's only going to set the description.

@mrworf
Copy link
Owner

mrworf commented Mar 21, 2017

Latest version should work now.

@Jonny007-MKD
Copy link
Contributor Author

Hey, it is indeed working :)
Now my inventory looks like this:

QGxSSyaOpEtOnT9UUQlw | 20170328-100633-16524 | 2017-03-28T11:58:18Z |
dKmaRnlUwFiIZYy_Wb_w | 20170328-100633-16524 | 2017-03-28T11:58:20Z | 
SOMqueK7vJS19zpsCIgQ | 20170328-100633-16524 | 2017-03-28T11:58:20Z | 
mNpDtkWDMJT7kX5mg_sg | 20170328-100633-16524 | 2017-03-28T11:58:21Z | 
RxpNX9cp84dxj4TYMlLQ | 20170328-100633-16524 | 2017-03-28T11:58:21Z | 
cNiqEgBfOUxD8iJSWGuQ | 20170328-100633-16524 | 2017-03-28T11:58:22Z |

Do you know whether Glacier remembers the file extension? Or can we afterwards automatically figure out which item is what?

@mrworf
Copy link
Owner

mrworf commented Mar 29, 2017

Odd, there should be a description AND a filename, not just the description

@Jonny007-MKD
Copy link
Contributor Author

Jonny007-MKD commented Mar 29, 2017 via email

@mrworf
Copy link
Owner

mrworf commented Mar 29, 2017

If you can, try downloading the smallest one and see if you get the actual filename. Otherwise we have a slight issue. Should be able to reconstitute the extension based on header in file, but I'd rather not do that

@Jonny007-MKD
Copy link
Contributor Author

Jonny007-MKD commented Mar 30, 2017 •
edited

Here is what I tried:

$ glacier-cmd ... download Music-Backup 61ZVotZte1tIZF_J0waSzobzVfFmH-Zj0eThIN42f2Q9LKTHtR5Chk-wdAZbDJenGQFFaE-CVqCMSYXBeQXlkOmtk_JMfWqklX3CHDhL4ZJ7do_HQVJI4GmNpDtkWDMJT7kX5mg_sg
334d4a393a19356b740e36227013a2069839066c  20170328-100633-16524.tar.bz2.gpg.sig.par2.sig
2abedfe7072338b2c48d532f49ada07cf4f66f91  20170328-100633-16524.tar.bz2.gpg.sig.vol07+8.par2.sig
87129da51fd2d5907ba8b3f1040d697e9fe9b3fe  20170328-100633-16524.tar.bz2.gpg.sig.vol03+4.par2.sig
2c13150e0b9fb51a04bd7fd937174e2406de8790  20170328-100633-16524.json.gpg.asc
[...]
Wrote 731.0 bytes. Rate 2.48 KB/s.00%). Rate 3.50 MB/s, average 2.50 KB/s, ETA 06:34:13.

Seems like I downloaded the list with filenames, whoever created it. So manually we could download all files with the same description, parse this special file and use the hashes to get the file names.

@mrworf
Copy link
Owner

mrworf commented Mar 30, 2017

That's actually iceshelf. To avoid exposing what's in your archive but still quickly check it's "OK" and that you have all files, it creates a .lst file which can be used with shasum. But I'm none too happy with the loss of filenames. That was not the intention of this change (beats himself over the head for not confirming this).

After some more digging, it seems that description and filename are one and the same here and most likely it's just for glacier-cmd's own benefit it has both. This makes it quite useless though. I always assumed the filename would be stored separately, but I guess not.

So should I revert to oiriginal behavior? I can remove the path of the file and just store the name itself, and since it's all prefixed with the ID of the archive, it should still be fine.

Thoughts?

@Jonny007-MKD
Copy link
Contributor Author

I totally agree with you, and I'm fine with your suggestion :) The path is quite useless, but with the filename we get the ID and the extension(s).

@mrworf
Copy link
Owner

mrworf commented Mar 30, 2017

Alright, cool, I'll fix that tonight (running on PST time here) so expect an update in ~10hrs.

Btw, if you want a more direct communication, send me an email and I'll invite you to the slack channel

@Jonny007-MKD
Copy link
Contributor Author

Any news on this? :)

@mrworf
Copy link
Owner

mrworf commented Apr 4, 2017

Sorry for lack of update, I'm just running a test to confirm that it does indeed upload the expected data to glacier before I push the change. Should be done in a few hours (waiting for inventory ...zzzzz)

@mrworf
Copy link
Owner

mrworf commented Apr 5, 2017

So I'm having some issues right now getting any sensible results from this. I see my files being uploaded, but I cannot make glacier-cmd's inventory call actually reflecting this. Even when I force it with --refresh it still lists old content from 2015. Have you seen this issue?

@Jonny007-MKD
Copy link
Contributor Author

Jonny007-MKD commented Apr 5, 2017 via email

@mrworf
Copy link
Owner

mrworf commented Apr 5, 2017

depressing :( oh well, it does work, I've seen it upload and issue the correct commands, so I'm just going to cleanup and then push it.

@mrworf
Copy link
Owner

mrworf commented Apr 5, 2017

I wish github had a messaging system. I have a slack for this already, if you send me your email to github@sensenet.nu I'll invite you.

@mrworf
Copy link
Owner

mrworf commented Apr 5, 2017

Alright, it works, and I also found the issue in glacier-cmd (for inventory), so it might be time to fork and maintain that too since it doesn't seem like the original owner have time anymore.

@Jonny007-MKD
Copy link
Contributor Author

Great, thanks! :) If you fork glacier-cmd I'll happily follow you!

@mrworf
Copy link
Owner

mrworf commented Apr 6, 2017

We'll see :) Might do it for this fix at least. Closing this issue now.

@mrworf mrworf closed this as completed Apr 6, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Jonny007-MKD @mrworf