Library layer - first part #11

ppolewicz · 2015-12-12T23:11:31Z

This PR is WIP (created for easy early review). Please do not merge it before it is ready.

This will eventually implement #4.

Task list:

Unfinished items will be done in another PR

…t_buckets()

ppolewicz · 2015-12-12T23:20:52Z

@bwbeach please review to confirm that our views are still aligned. list_buckets now has a library interface and a console command handler interface which depends on the library interface.

Console version should not be affected.

Example usage of library:

>>> import b2
>>> api = b2.B2Api(b2.StoredAccountInfo())
>>> for b in api.list_buckets():
...     print b.name
... 
test234
>>> b = api.get_bucket_by_name('foobar')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "b2.py", line 305, in get_bucket_by_name
    raise NonExistentBucket(bucket_name)
b2.NonExistentBucket: No such bucket: foobar
>>>

ppolewicz · 2015-12-15T00:19:18Z

@bwbeach this is blocking for me: I don't want to put a few dozen hours into it without first making sure that my way of doing it is fine for you (and will get merged). Could you please take a quick look at it by tomorrow EOD?

bwbeach · 2015-12-15T00:56:07Z

Pawel,

Sorry for the delay. Just got back traveling this weekend.

Your changes look good, and I like the direction you're taking. Just a few comments:

What's the reason for mapping bucket types to 1001 and 1002? It might be easier when debugging just to use the strings names internally.
I like the exception hierarchy.
I don't yet see where you're going with the class hierarchy AbstractBucket/UnknownTypeBucket/Bucket.
The layering is looking good.
Your coding style is clean and readable.

BrianB

ppolewicz · 2015-12-15T23:09:29Z

The thing with bucket types is a generic pattern for enum in python. You might be right, it may not be very useful here, I'll make a note to look at it again at the end and remove it if it is still not useful.

Bucket types is a defense against user having to check if fields of bucket are None everywhere, so that any operation on bucket is contained in the class rather than splattered in every place someone needs to use it... However it is clear that you are working with b2 much longer than me - I can not think of any operations that would need to be extracted like that. I added a note to review and simplify that as well.

Than you very much for the review, now I can proceed.

* origin/master: Add retry logic for uploads, to upload to a different storage pod if the chosen one was busy. Bump version number. Add comment about version number conventions. Conflicts: b2

* origin/master: Bump version number. Check that file is not too big before trying to upload. Compute sha1 of uploaded file only once, even when retrying upload. Support external SHA1 sum Conflicts: b2

ppolewicz · 2015-12-26T01:05:39Z

@bwbeach

api = b2.B2Api()
bucket = api.get_bucket_by_name('foobar')
api.delete_bucket(bucket)

or

api = b2.B2Api()
bucket = api.get_bucket_by_name('foobar')
bucket.delete()

? I think the first version is better, because bucket.delete() might be confused between deleting a file and deleting a bucket, while api.delete_bucket(bucket) can not be confused with anything.

This makes it easier to use the api, for example: api = B2Api()

bwbeach · 2015-12-26T17:31:34Z

api = b2.B2Api(b2.StoredAccountInfo())
bucket = api.get_bucket_by_name('foobar')
api.delete_bucket(bucket)
or

api = b2.B2Api(b2.StoredAccountInfo())
bucket = api.get_bucket_by_name('foobar')
bucket.delete()
? I think the first version is better, because bucket.delete() might be confused between deleting a file and deleting a bucket, while api.delete_bucket(bucket) can not be confused with anything.

I agree. I think api.delete_bucket(bucket) is clearer, and I like keeping the name "delete_bucket" consistent with the B2 web API.

BrianB

ppolewicz · 2015-12-30T02:42:20Z

@bwbeach why does b2_delete_file_version api call accept a file name parameter? Isn't the file id sufficient to identify a file in the b2 storage?

This is needed for bucket methods which need to call the API

* origin/master: Rename class stream_with_progress to StreamWithProgress, following PEP 8 conventions. Update shebang line to work with default OSX Python installation. Remove the option to disable progress bar during uploads Add a progress bar for downloads Add a progress bar for uploads Sort imports Conflicts: b2

bwbeach · 2015-12-30T06:10:56Z

Passing the file name into b2_delete_file_version makes the implementation easier. It's quite possible to do without it, but saves us a step, and makes things faster, if the name is passed in. We are considering removing that requirement in a future version of the API.

Each command will now print helpful info if you get its arguments wrong, without spewing out info about all commands.

* master: Add --quiet option to upload_file. More complete tests. Add tests for all options to "b2 ls". Add tester for b2: test_b2_command_line.py. Conflicts: b2

ppolewicz · 2016-01-03T03:22:44Z

@bwbeach #20 modifies the same code that I am going to modify. As you intend to accept #20, I'll merge it when working on #11 to avoid a massive conflict.

ppolewicz · 2016-01-03T18:38:52Z

@bwbeach I am removing print 'Getting upload URL...' as it breaks abstraction in a major way and I do not think it is very important to communicate this to the user. We can add it back later if it is needed after all.

bwbeach · 2016-01-03T19:29:53Z

Sounds good.

On Jan 3, 2016, at 08:38, Paweł Polewicz notifications@github.com wrote:

@bwbeach https://github.com/bwbeach I am removing print 'Getting upload URL...' as it breaks abstraction in a major way and I do not think it is very important to communicate this to the user. We can add it back later if it is needed after all.

—
Reply to this email directly or view it on GitHub #11 (comment).

bwbeach · 2016-01-03T19:30:58Z

How close are you on #11? If you're close, I could wait on #20.

I'll go ahead an pull in #20 later today if I don't hear from you.

On Jan 2, 2016, at 17:22, Paweł Polewicz notifications@github.com wrote:

@bwbeach https://github.com/bwbeach #20 #20 modifies the same code that I am going to modify. As you intend to accept #20 #20, I'll merge it when working on #11 #11 to avoid a massive conflict.

—
Reply to this email directly or view it on GitHub #11 (comment).

ppolewicz · 2016-01-03T22:12:41Z

I am close. I will finish tonight.
Changes in upload_file are are hard. Everything is glued together to everything in there.

It is my fault that #20 will have conflicts and I can see that you intend to merge it in, so in order to avoid putting additional burden on @cederberg, I will try to use git magic to merge it and resolve the conflicts myself. It has to be in the right order with my commits or otherwise the conflict will be massive.

ppolewicz · 2016-01-03T22:33:53Z

@bwbeach there is code to cope with the fact that the API has returned a dictionary after uploading a file, but the dictionary did not contain the fileId key. I suspect that an early version of b2 api did that. Can I remove it now and assume fileId will be returned properly?

bwbeach · 2016-01-03T22:52:10Z

I think that was an unclear way of checking whether an error was returned, but there are better checks for it now. Feel free to remove it and clean up the code.

On Jan 3, 2016, at 12:33, Paweł Polewicz notifications@github.com wrote:

@bwbeach https://github.com/bwbeach there is code to cope with the fact that the API has returned a dictionary after uploading a file, but the dictionary did not contain the fileId key. I suspect that an early version of b2 api did that. Can I remove it now and assume fileId will be returned properly?

—
Reply to this email directly or view it on GitHub #11 (comment).

bwbeach · 2016-01-04T01:14:29Z

I'll wait until you're done.

BrianB

On Jan 3, 2016, at 12:12, Paweł Polewicz notifications@github.com wrote:

I am close. I will finish tonight.
Changes in upload_file are are hard. Everything is glued together to everything in there.

It is my fault that #20 #20 will have conflicts and I can see that you intend to merge it in, so in order to avoid putting additional burden on @cederberg https://github.com/cederberg, I will try to use git magic to merge it and resolve the conflicts myself. It has to be in the right order with my commits or otherwise the conflict will be massive.

—
Reply to this email directly or view it on GitHub #11 (comment).

ppolewicz · 2016-01-04T01:21:41Z

I started work on upload_file, which required refactoring of error handling, which required refactoring of retry mechanic, which required refactoring of OpenUrl, and so on. One thing led to another and according to git diff --stat, 465 LOC were added/deleted. It is 2 AM and my programming skills are getting weaker.

IMO it is not a good idea to merge those new changes with the previous changes. Therefore lets execute plan B: merge it as it currently is and do the remaining items in other pull requests.

I will coordinate the merging and testing of #20 in its pull request.

ppolewicz · 2016-01-04T01:24:06Z

just a moment, test failure

ppolewicz · 2016-01-04T01:27:54Z

It was a bad git add, I fixed it. Ready for merge.

Library layer

even without tqdm.

ppolewicz added 2 commits December 13, 2015 00:01

Sort imports

217abc7

Create library access skeleton and split console from library for lis…

9ffd9dd

…t_buckets()

Implement B2Api.get_bucket_by_id() and IdOnlyBucket

6cae1b3

ppolewicz added 5 commits December 16, 2015 00:10

Add TODO comments

a82f7c5

Merge remote-tracking branch 'origin/master' into library_layer

6b33114

* origin/master: Add retry logic for uploads, to upload to a different storage pod if the chosen one was busy. Bump version number. Add comment about version number conventions. Conflicts: b2

Implement B2Api.create_bucket()

fa89836

Merge remote-tracking branch 'origin/master' into library_layer

0add7dd

* origin/master: Bump version number. Check that file is not too big before trying to upload. Compute sha1 of uploaded file only once, even when retrying upload. Support external SHA1 sum Conflicts: b2

Implement B2Api.delete_bucket()

6fb8aa3

ppolewicz added 3 commits December 26, 2015 02:29

Make StoredAccountInfo default authorization handler for B2Api

6249da0

This makes it easier to use the api, for example: api = B2Api()

Refactor ConsoleTools slightly

a856d0a

Implement B2Api.make_url()

0bb4430

ppolewicz force-pushed the library_layer branch from fd38c4f to 0bb4430 Compare December 26, 2015 02:12

Rename B2Api property from "authorizator" to "account_info"

ac79edc

ppolewicz added 5 commits December 30, 2015 05:04

Cleanup and merge of file info DAO classes

561a77e

Cleanup of comment

998b307

Add api to bucket constructor

bf04b71

This is needed for bucket methods which need to call the API

Implement AbstractBucket.hide_file()

2851023

This was referenced Dec 30, 2015

Add a progress bar #21

Merged

Added new 'sync' action for easy local <-> remote copying #20

Merged

ppolewicz referenced this pull request Dec 31, 2015

Convert the rest of the commands to use parseargs.

31c51ed

Each command will now print helpful info if you get its arguments wrong, without spewing out info about all commands.

ppolewicz referenced this pull request Dec 31, 2015

Switch "b2 ls" to use argparse.

5f0cff5

ppolewicz mentioned this pull request Dec 31, 2015

Need --quiet option to make output easy to parse. #23

Closed

ppolewicz added 8 commits January 2, 2016 20:05

Merge branch 'master' into library_layer

43a23fb

* master: Add --quiet option to upload_file. More complete tests. Add tests for all options to "b2 ls". Add tester for b2: test_b2_command_line.py. Conflicts: b2

Drop getUploadUrl() accidentally copied from java interface

ebe1888

Implement B2Api.delete_file_version()

0b70a1a

Remove FileInfo class

8213a73

Implement Bucket.set_type() / ConsoleTool.update_bucket(args)

0281881

Remove useless BucketType enum

c868516

Remove TODO comments

2c67bf7

Implement InMemoryCache.save_bucket()

2019ac0

Implement ConsoleTool.clear_account()

c3050f0

ppolewicz force-pushed the library_layer branch from 2c636c6 to c3050f0 Compare January 4, 2016 01:01

Fix upload_file() (it is not in ConsoleTool yet)

7795e38

ppolewicz changed the title ~~Library layer~~ Library layer - first part Jan 4, 2016

bwbeach added a commit that referenced this pull request Jan 4, 2016

Merge pull request #11 from ppolewicz/library_layer

c16c223

Library layer

bwbeach merged commit c16c223 into Backblaze:master Jan 4, 2016

ppolewicz deleted the library_layer branch January 5, 2016 23:43

ppolewicz referenced this pull request Jan 7, 2016

Merge in conflicted changed to add progress reported

ede2529

even without tqdm.

ppolewicz mentioned this pull request Jan 25, 2016

using b2 command line tool as a library #4

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Library layer - first part #11

Library layer - first part #11

ppolewicz commented Dec 12, 2015

ppolewicz commented Dec 12, 2015

ppolewicz commented Dec 15, 2015

bwbeach commented Dec 15, 2015

ppolewicz commented Dec 15, 2015

ppolewicz commented Dec 26, 2015

bwbeach commented Dec 26, 2015

ppolewicz commented Dec 30, 2015

bwbeach commented Dec 30, 2015

ppolewicz commented Jan 3, 2016

ppolewicz commented Jan 3, 2016

bwbeach commented Jan 3, 2016

bwbeach commented Jan 3, 2016

ppolewicz commented Jan 3, 2016

ppolewicz commented Jan 3, 2016

bwbeach commented Jan 3, 2016

bwbeach commented Jan 4, 2016

ppolewicz commented Jan 4, 2016

ppolewicz commented Jan 4, 2016

ppolewicz commented Jan 4, 2016

Library layer - first part #11

Library layer - first part #11

Conversation

ppolewicz commented Dec 12, 2015

ppolewicz commented Dec 12, 2015

ppolewicz commented Dec 15, 2015

bwbeach commented Dec 15, 2015

ppolewicz commented Dec 15, 2015

ppolewicz commented Dec 26, 2015

bwbeach commented Dec 26, 2015

ppolewicz commented Dec 30, 2015

bwbeach commented Dec 30, 2015

ppolewicz commented Jan 3, 2016

ppolewicz commented Jan 3, 2016

bwbeach commented Jan 3, 2016

bwbeach commented Jan 3, 2016

ppolewicz commented Jan 3, 2016

ppolewicz commented Jan 3, 2016

bwbeach commented Jan 3, 2016

bwbeach commented Jan 4, 2016

ppolewicz commented Jan 4, 2016

ppolewicz commented Jan 4, 2016

ppolewicz commented Jan 4, 2016