using b2 command line tool as a library

[ppolewicz]

I would like to implement something in python using b2 and its command line tool. Current implementation already lets me import it (after I symlink it to b2.py, that is) and call functions on it, which is good, however for any application which is not console-based or requires some resiliency against errors, it is impossible to use due to the current architecture. It is also not possible to test it automatically in isolation. Clearly the current codebase is already designed with integration with other python code - at least partially.

The problem is with two layers being mixed with each other. One layer is what performs operations on the backend (let's call it controller) and the other is what displays the retrieved information (let's call it view). For example, if I would like to create a GUI application which would present the result of ls, I can't, because the function uses print to return results to the user. Here the layers should be divided so that the backend function returns an iterator of entries (objects), which would then be printed by the view function. If someone would need to call a library equivalent of ls, he would call the backend one and then display the result in the GUI (or web page or whatever he is trying to build).

Another example is that if something does not exist or there is some other problem, current tool calls sys.exit(). If the user supplies invalid data to the gui application, as an application developer I want to display an error message, not shutdown the whole program. Therefore I'd expect the library functions to raise an exception on error, so that it can be handled properly.

It is possible to implement most of the functionality of b2 command line tool again in a way which has properly separated layers and allows for console, gui, web page and other types of outputs, as well as for automated testing in isolation.

On the other hand, the same changes could be developed here in this repository, with keeping most of the interface as it is. To be precise, I'd contribute most (if not all) of the code and tests required to do this properly. As it is a larger amount of work, I'd like to discuss it first here, so if my view and views of official maintainers are not aligned, it can be compared before much work is put into development.

Please comment the above.

[bwbeach]

Making a library or class for programmatic access to the B2 APIs needs to be done, and if you'd like to help out with that, it would be great.

We currently have a Java class that we use for most of our testing. Having a Python interface as well would be helpful.

This is what the Java interface looks like:

/**
 * B2Api provides a stateless API for performing b2 operations.
 * Please read the <a href="https://www.backblaze.com/b2/docs/">B2 documentation</a> for details.
 * Note that this class takes care of percent-encoding your filenames and header values, so *not* encode them yourself.
 *
 * Copyright 2015, Backblaze, Inc. All rights reserved.
 */
public interface B2Api {
    /**
     *
     * @param accountId the account we're trying to authenticate
     * @param applicationKey that account's applicationKey
     * @return returns details needed to use b2, if authorization succeeds.
     * @throws WebApiError
     */
    AccountAuthorization authorizeAccount(String accountId,
                                          String applicationKey) throws WebApiError;

    /**
     *
     * @param accountAuth the authorization details for the account you want to use
     * @param bucketName the name of the bucket you're asking to create
     * @param bucketType the permission type for the bucket you're asking to create
     * @return a Bucket object with details, if creation works
     * @throws WebApiError
     */
    Bucket createBucket(AccountAuthorization accountAuth,
                        String bucketName,
                        Bucket.BucketType bucketType) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param bucketId the id of the bucket you're asking to delete
     * @return info about that bucket, if deletion works
     * @throws WebApiError
     */
    Bucket deleteBucket(AccountAuthorization accountAuth,
                        String bucketId) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param fileName the name of the file you're asking to delete
     * @param fileId the id of the file (version) you're asking to delete
     * @return info about the file version that was deleted, if deletion succeeds.
     * @throws WebApiError
     */
    FileNameAndId deleteFileVersion(AccountAuthorization accountAuth,
                                    String fileName,
                                    String fileId) throws WebApiError;

    /**
     * This attempts a download of the specified file.  It always includes the
     * account authorization with the request.
     *
     * @param accountAuth the authorization details for the account you want to use
     * @param fileId the id of the file (version) you're asking to download
     * @param contentHandler a handler for the downloaded content
     * @throws WebApiError
     */
    void downloadFileById(AccountAuthorization accountAuth,
                          String fileId,
                          BzWebClient.ContentHandler contentHandler) throws WebApiError;

    /**
     * This attempts a download of the specified file.  It always includes the
     * account authorization with the request.
     *
     * @param accountAuth the authorization details for the account you want to use
     * @param fileName the name of the file you're asking to download the latest version of.
     * @param contentHandler a handler for the downloaded content
     * @throws WebApiError
     */
    void downloadFileByName(AccountAuthorization accountAuth,
                            String bucketName,
                            String fileName,
                            BzWebClient.ContentHandler contentHandler) throws WebApiError;

    /**
     * This attempts a download of the specified file.  It never includes the
     * account authorization with the request.  To emphasize that, you only pass
     * in the downloadUrl, not a full AccountAuthorization.
     *
     * @param baseDownloadUrl the downloadUrl you got from an authorize_account call.
     * @param fileId the id of the file (version) you're asking to download
     * @param contentHandler a handler for the downloaded content
     * @throws WebApiError
     */
    void downloadPublicFileById(String baseDownloadUrl,
                                String fileId,
                                BzWebClient.ContentHandler contentHandler) throws WebApiError;
    /**
     * This attempts a download of the specified file.  It never includes the
     * account authorization with the request.  To emphasize that, you only pass
     * in the downloadUrl, not a full AccountAuthorization.
     *
     * @param baseDownloadUrl the downloadUrl you got from an authorize_account call.
     * @param fileName the name of the file you're asking to download the latest version of.
     * @param contentHandler a handler for the downloaded content
     * @throws WebApiError
     */
    void downloadPublicFileByName(String baseDownloadUrl,
                                  String bucketName,
                                  String fileName,
                                  BzWebClient.ContentHandler contentHandler) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param fileId the id of the file (version) you're asking for information about
     * @return file info for the specified file, if it exists.
     * @throws WebApiError
     */
    FileInfoResponse getFileInfo(AccountAuthorization accountAuth,
                                 String fileId) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param bucketId the id of the bucket you're asking to upload to
     * @return the info you need to perform an upload.
     * @throws WebApiError
     */
    GetUploadUrlResponse getUploadUrl(AccountAuthorization accountAuth,
                                      String bucketId) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param bucketId the bucket containing the file you're asking to hide.
     * @param fileName the name of the file you're asking to hide.
     * @throws WebApiError
     */
    FileNameInfo hideFile(AccountAuthorization accountAuth,
                          String bucketId,
                          String fileName) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account whose buckets you're asking to list
     * @throws WebApiError
     */
    ListBucketsResponse listBuckets(AccountAuthorization accountAuth) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param bucketId the bucket you're asking to list
     * @param startFileNameOrNull the starting point for the search.  if not specified, starts with the first name in the bucket.
     * @param maxFileCountOrNull the maximum number of answers to return.  if not specified, will use the service's default.
     * @return FileNameAndInfos for files that match your list criteria.
     * @throws WebApiError
     */
    ListFileNamesResponse listFileNames(AccountAuthorization accountAuth,
                                        String bucketId,
                                        String startFileNameOrNull,
                                        Integer maxFileCountOrNull) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param bucketId the bucket you're asking to list
     * @param maxFileCountOrNull the maximum number of answers to return.  if not specified, will use the service's default.
     * @param startFileNameOrNull the starting point for the search.  if not specified, starts with the first name in the bucket.
     * @param startFileIdOrNull the starting point for the search.
     *                          if non-null, startFileNameOrNull must also be non-null.
     *                          if non-null, adjusts starting point as described in web documentation.
     * @return FileNameAndInfos for files that match your list criteria.
     * @throws WebApiError
     */
    ListFileVersionsResponse listFileVersions(AccountAuthorization accountAuth,
                                              String bucketId,
                                              Integer maxFileCountOrNull,
                                              String startFileNameOrNull,
                                              String startFileIdOrNull) throws WebApiError;

    /**
     * @param accountAuth the authorization details for the account you want to use
     * @param bucketId the bucket you're asking update
     * @param bucketType the permission type you're asking to change to
     * @return a description of the bucket that was updated
     * @throws WebApiError
     */
    Bucket updateBucket(AccountAuthorization accountAuth,
                        String bucketId,
                        Bucket.BucketType bucketType) throws WebApiError;

    /**
     * @param uploadUrlResponse the upload url response from a previous call to getUploadUrl() for the targeted bucket.
     * @param fileName the name the file should have
     * @param mimeTypeOrNull if null or "b2/x-auto", b2 will assign a mime-type.
     * @param contentBytes the contents of the file
     * @param contentSha1 the sha1 of the contentBytes.
     * @param extraHeaderNamesAndValues an even number of strings which are interpreted
     *                                  as alternating headerName and headerValue pairs, eg:
     *                                  "X-Bz-Info-Color", "blue", "X-Bz-Info-Flower", "Calla Lilly"
     *                                  (do not encode the values for transport.  this class does that.)
     * @return info about the file if it was successfully uploaded.
     * @throws WebApiError
     */
    FileInfoResponse uploadFile(GetUploadUrlResponse uploadUrlResponse,
                                String fileName,
                                String mimeTypeOrNull,
                                byte[] contentBytes,
                                String contentSha1,
                                String... extraHeaderNamesAndValues) throws WebApiError;
}

[ppolewicz]

This is almost exactly how I imagined it should look like.

How do you want to do it? Should make a big pull request which would then be updated as I rewrite the core and front methods gradually, while you can observe / review the process and then after all review comments are applied, it will be merged?

[bwbeach]

That sounds great. I look forward to working with you on it.

[bwbeach]

BTW, in the Java code we have another layer on top of the B2Api that keep track of bucket IDs, and all of the auth tokens. It's not quite what the b2 command-line program wants, because the command-line has its own way of managing and persisting the auth tokens.

Might be worth keeping in mind for the future, as another possible layer in the Python code.

/**
 * B2PersonalApi provides a stateful API for performing b2 operations.
 * It should only be used in one thread.  It differs from B2Api in
 * that it manages the AccountAuthorization and bucket upload urls
 * automatically.
 *
 * Please read the <a href="https://www.backblaze.com/b2/docs/">B2 documentation</a> for details.
 * Note that this class takes care of percent-encoding your filenames and header values,
 * so do *not* encode them yourself.
 *
 * You will provide your accountId and applicationKey when you construct an object
 * that implements this class.
 *
 * Copyright 2015, Backblaze, Inc. All rights reserved.
 */
public interface B2PersonalApi {

    /**
     * @param bucketName the name of the bucket you're asking to create
     * @param bucketType the permission type for the bucket you're asking to create
     * @return a Bucket object with details, if creation works
     * @throws WebApiError
     */
    Bucket createBucket(String bucketName,
                        Bucket.BucketType bucketType) throws WebApiError;

    /**
     * @param bucketId the id of the bucket you're asking to delete
     * @return info about that bucket, if deletion works
     * @throws WebApiError
     */
    Bucket deleteBucket(String bucketId) throws WebApiError;

    /**
     * @param fileName the name of the file you're asking to delete
     * @param fileId the id of the file (version) you're asking to delete
     * @return info about the file version that was deleted, if deletion succeeds.
     * @throws WebApiError
     */
    FileNameAndId deleteFileVersion(String fileName,
                                    String fileId) throws WebApiError;

    /**
     * This attempts a download of the specified file.  It always includes the
     * account authorization with the request.
     *
     * @param fileId the id of the file (version) you're asking to download
     * @param contentHandler a handler for the downloaded content
     * @throws WebApiError
     */
    void downloadFileById(String fileId,
                          BzWebClient.ContentHandler contentHandler) throws WebApiError;

    /**
     * This attempts a download of the specified file.  It always includes the
     * account authorization with the request.
     *
     * @param fileName the name of the file you're asking to download the latest version of.
     * @param contentHandler a handler for the downloaded content
     * @throws WebApiError
     */
    void downloadFileByName(String bucketName,
                            String fileName,
                            BzWebClient.ContentHandler contentHandler) throws WebApiError;

    /**
     * @param fileId the id of the file (version) you're asking for information about
     * @return file info for the specified file, if it exists.
     * @throws WebApiError
     */
    FileInfoResponse getFileInfo(String fileId) throws WebApiError;

    /**
     * @param bucketId the id of the bucket you're asking to upload to
     * @return the info you need to perform an upload.
     * @throws WebApiError
     */
    GetUploadUrlResponse getUploadUrl(String bucketId) throws WebApiError;

    /**
     * @param bucketId the bucket containing the file you're asking to hide.
     * @param fileName the name of the file you're asking to hide.
     * @throws WebApiError
     */
    FileNameInfo hideFile(String bucketId,
                          String fileName) throws WebApiError;

    /**
     * @throws WebApiError
     */
    ListBucketsResponse listBuckets() throws WebApiError;

    /**
     * @param bucketId the bucket you're asking to list
     * @param startFileNameOrNull the starting point for the search.  if not specified, starts with the first name in the bucket.
     * @param maxFileCountOrNull the maximum number of answers to return.  if not specified, will use the service's default.
     * @return FileNameAndInfos for files that match your list criteria.
     * @throws WebApiError
     */
    ListFileNamesResponse listFileNames(String bucketId,
                                        String startFileNameOrNull,
                                        Integer maxFileCountOrNull) throws WebApiError;

    /**
     * returns a new iterator for the given bucketId, starting at the specified
     * fileName, if any.
     *
     * @param bucketId the bucket to iterate over
     * @param startFileNameOrNull the starting point for the search.  see listFileNames.
     * @return an iterator which might throw a RuntimeException from next() if
     *         it has trouble fetching additional answers.
     */
    default Iterable<FileNameInfo> fileNames(String bucketId,
                                             String startFileNameOrNull,
                                             Integer maxFileCountOrNull) {
        return () -> {
            try {
                return new PersonalFileNamesIterator(
                        B2PersonalApi.this,
                        bucketId,
                        maxFileCountOrNull,
                        startFileNameOrNull);
            } catch (WebApiError e) {
                throw new RuntimeException("failed to create iterator: " + e.getMessage(), e);
            }
        };
    }

    /**
     * @param bucketId the bucket you're asking to list
     * @param maxFileCountOrNull the maximum number of answers to return.  if not specified, will use the service's default.
     * @param startFileNameOrNull the starting point for the search.  if not specified, starts with the first name in the bucket.
     * @param startFileIdOrNull the starting point for the search.
     *                          if non-null, startFileNameOrNull must also be non-null.
     *                          if non-null, adjusts starting point as described in web documentation.
     * @return FileNameAndInfos for files that match your list criteria.
     * @throws WebApiError
     */
    ListFileVersionsResponse listFileVersions(String bucketId,
                                              Integer maxFileCountOrNull,
                                              String startFileNameOrNull,
                                              String startFileIdOrNull) throws WebApiError;

    /**
     * returns a new iterator for the given bucketId, starting at the specified
     * fileName, if any.
     *
     * @param bucketId the bucket to iterate over
     * @param startFileNameOrNull the starting point for the search.  see listFileNames.
     * @return an iterator which might throw a RuntimeException from next() if
     *         it has trouble fetching additional answers.
     */
    default Iterable<FileNameInfo> fileVersions(String bucketId,
                                                Integer maxFileCountOrNull,
                                                String startFileNameOrNull,
                                                String startFileIdOrNull) {
        return () -> {
            try {
                return new PersonalFileVersionsIterator(
                        B2PersonalApi.this,
                        bucketId,
                        maxFileCountOrNull,
                        startFileNameOrNull,
                        startFileIdOrNull);
            } catch (WebApiError e) {
                throw new RuntimeException("failed to create iterator: " + e.getMessage(), e);
            }
        };
    }


    /**
     * @param bucketId the bucket you're asking update
     * @param bucketType the permission type you're asking to change to
     * @return a description of the bucket that was updated
     * @throws WebApiError
     */
    Bucket updateBucket(String bucketId,
                        Bucket.BucketType bucketType) throws WebApiError;

    /**
     * @param bucketId the bucket you want to upload to.
     * @param fileName the name the file should have
     * @param mimeTypeOrNull if null or "b2/x-auto", b2 will assign a mime-type.
     * @param contentBytes the contents of the file
     * @param contentSha1 the sha1 of the contentBytes.
     * @param extraHeaderNamesAndValues an even number of strings which are interpreted
     *                                  as alternating headerName and headerValue pairs, eg:
     *                                  "X-Bz-Info-Color", "blue", "X-Bz-Info-Flower", "Calla Lilly"
     *                                  (do not encode the values for transport.  this class does that.)
     * @return info about the file if it was successfully uploaded.
     * @throws WebApiError
     */
    FileInfoResponse uploadFile(String bucketId,
                                String fileName,
                                String mimeTypeOrNull,
                                byte[] contentBytes,
                                String contentSha1,
                                String... extraHeaderNamesAndValues) throws WebApiError;


    /**
     * @param bucketName name of the bucket we're interested in.
     * @return info about the bucket in this account with that name or null if there isn't a bucket with that name in this account.
     * @throws WebApiError
     */
    Bucket getBucketByName(String bucketName) throws WebApiError;
}

[ppolewicz]

One more thing. Currently there are some options like -dev, -staging, -production, -contentType <contentType> and -info <key>=<value>, but the convention in linux tool (and standard library support in python) is to use double dash, so --production and so on. It would be easy to correct this when refactoring to add the layers we already mentioned.

Can I fix it?

Is it ok if I do both of those in one go?

[bwbeach]

Let me go ahead and to it.

I just changed the syntax of the command line yesterday, and we haven't published the new version on the web site yet. If I make the change now, it's just one thing to put in the release notes.

• Brian

On Dec 4, 2015, at 10:04, Paweł Polewicz notifications@github.com wrote:

One more thing. Currently there are some options like -dev, -staging, -production, -contentType and -info =, but the convention in linux tool (and standard library support in python) is to use double dash, so --production and so on. It would be easy to correct this when refactoring to add the layers we already mentioned.

Can I fix it?

Is it ok if I do both of those in one go?

—
Reply to this email directly or view it on GitHub #4 (comment).

[ppolewicz]

Ok, another question then. Do we still want to keep everything in one file?

[bwbeach]

I'd like to stick with one file, unless it becomes a big problem. We like having a single file to download from the web site that is the command-line tool.

At some point it will get big enough, or complicated enough, that it's ungainly, but it would be nice if that's off in the future.

[ppolewicz]

Can we make it so that the sources are spread into many files, which are then magically put into one file during a build process, so that we can have both a tidy project and downloadable single file solution for people that just need it to work?

That is for the future though, as you say. I'll stick with the one file for now.

[bwbeach]

Can we make it so that the sources are spread into many files, which are then magically put into one file during a build process, so that we can have both a tidy project and downloadable single file solution for people that just need it to work?

Sounds good.
That is for the future though, as you say. I'll stick with the one file for now.

Cool.

The other option for easy installation, which AWS uses for its command-line tool, is to make it available via pip.

[ppolewicz]

Ok.

I'll start working on #4 this weekend.

[ppolewicz]

@bwbeach do you need to keep compatibility with Python 2.6?

[bwbeach]

No. Python 2.7 is fine.

On Dec 12, 2015, at 13:11, Paweł Polewicz notifications@github.com wrote:

@bwbeach https://github.com/bwbeach do you need to keep compatibility with Python 2.6?

—
Reply to this email directly or view it on GitHub #4 (comment).

[ferricoxide]

@bwbeach: Be aware that Enterprise Linux 6 (Red Hat, CentOS, SciLin, etc.) do not include python 2.7 support in the standard RPM repos.

[bwbeach]

Well that's annoying.

On Dec 13, 2015, at 09:05, Thomas H Jones II notifications@github.com wrote:

@bwbeach https://github.com/bwbeach: Be aware that Enterprise Linux 6 (Red Hat, CentOS, SciLin, etc.) do not include python 2.7 support in the standard RPM repos.

—
Reply to this email directly or view it on GitHub #4 (comment).

[ferricoxide]

Tell me about it.

I support a few customers that have several thousand EL6-based systems and they're unlikely to be upgrading any of them until at least mid-2016. Their enterprise IA groups are waiting on SCAP-related guidance for EL7 to get finalized before allowing migrations of production services to EL7. Python 2.7 exists in the SCL repositories, but most of my customers haven't them approved for production.

Enterprise Linux customers tend to be cautious with production systems.

[bwbeach]

Thanks for the useful info. Sounds like we should stick with supporting 2.6. :-/

• BrianB

On Dec 13, 2015, at 10:03, Thomas H Jones II notifications@github.com wrote:

Tell me about it.

I support a few customers that have several thousand EL6-based systems and they're unlikely to be upgrading any of them until at least mid-2016. Their enterprise IA groups are waiting on SCAP-related guidance for EL7 to get finalized before allowing migrations of production services to EL7. Python 2.7 exists in the SCL repositories, but most of my customers haven't them approved for production.

Enterprise Linux customers tend to be cautious with production systems.

—
Reply to this email directly or view it on GitHub #4 (comment).

[ppolewicz]

@ferricoxide actually RHEL6 is exactly why I asked the question in the first place. Thank you for helping to clear it up.

[ppolewicz]

Last item from #11 is in. Shall we close #4?

[bwbeach]

Yes. It's time to close this one. Thanks for all the work!

[shuhaowu]

How stable is the API? Can I use it in an application?

Edit: I ask because there are currently no generated docs anywhere. I'm relying on reading the source code and referring to the Java spec above.

[ppolewicz]

Don't rely on the Java spec, we didn't follow it strictly.

I don't think the API has changed in an incompatible way even once since it was born, so the stability is kind of infinite.

In my opinion using the api than doing things on your own - if b2 cloud API will change, we will compensate for it with our code, so that you don't have to.

As it is a rare event that we can actually talk, I'd like to ask what your usage will be. If you share that, we could design future features with taking your interest into account.

As for the documentation, I think nobody thought about it. Do you have any suggestion on what the docs should be like (an example from some other project perhaps)?

[shuhaowu]

As a follow up: I think I'll be just writing some request calls rather than using this library for now due to the lack of documentation. Furthermore, I think it would be beneficial for this library to have some simple unittests and maybe a refactor (library api directory + cmd line interface file) so the code is not 2kLoC all in one file. This would allow me to more easily inspect the library's implementation, by looking through the documentation and drilling down to the implementation by looking through directory listings rather than grepping through a single file. It is also somewhat difficult to find enough time to review a 2kLoC file.

A suggested library structure might be something like (obviously would require a little bit more work):

b2/
  bucket.py
  auth.py
  file.py
  ...
tests/
  b2/
    test_bucket.py
    test_auth.py

b2_cmd.py 

As for the documentation, I think nobody thought about it. Do you have any suggestion on what the docs should be like (an example from some other project perhaps)?

Usually python projects are documented via Sphinx. Usually it consts of a tutorial sort like page to demo/sell the project and then the API references for people active working on the project who needs to know about all the object.

A good example may be found here: http://basho.github.io/riak-python-client/index.html

Unfortunately the example above moved its tutorial to a separate page, but it links that in the top of main page.

As it is a rare event that we can actually talk, I'd like to ask what your usage will be. If you share that, we could design future features with taking your interest into account.

My usage case right now is extremely straightforward as I'm currently evaluating B2. Once a day, I will be generating a report of some sort and uploading a file on to B2. It then takes the link to the file and send it out over email. From as far as I can tell, I would need to:

b2_authorize_account
b2_get_upload_url
b2_upload_file

Then I should be able to construct the link to email.

I foresee be doing other things with B2, but that probably will not be relying on a Python client. I'll have to see about that.

In my opinion using the api than doing things on your own - if b2 cloud API will change, we will compensate for it with our code, so that you don't have to.

I definitely have this concern as well. Furthermore it is possible to use a library such that I can hide details about upload_url, api_url and such as obtained through the b2 http api, it will make the code simpler to read.

However, given the issues I have outlined above with regarding documentation and the difficulty of inspecting the code and the tasks I have in mind is very straightforward as outlined, I'll skip out on the library approach for now. I'll keep following the project and definitely will re-evaluate (or maybe contribute, depending on the needs) this project in the future.

[bwbeach]

I agree about having unit tests and refactoring into multiple source files. We were waiting for pip install before going to multiple files; that's done now, so the restructuring can proceed.

I also agree about Sphinx docs. They need to be written. Hoping to get that done relatively soon.