Write your own client

Jean-Francois SMIGIELSKI edited this page Nov 11, 2015 · 9 revisions

Ok, you are about to write your own OpenIO SDS client. This is nice, welcome in our community !

Foreword

First, you should consider reading OpenIO-Architecture. This is not mandatory but it will give you the keys to understand our choices.

Using OpenIO means handling objets referenced in container. You should at least be comfortable with our naming conventions, described at OpenIO-Object-Names.

Working with a client API brings your application into your OpenIO grid, while working with a gateway keeps the applicaiton aside of the grid. You need to know what you are doing, because you can do things really wrong.

Eventually you will want to test your code, then OpenIO-Local-Sandbox contains precious information that will help you setting up a local development environment.

General design

OpenIO has been initially designed for emails platforms: a lot of small files, organized into a lot of containers. The entities you will have to manage reflect this design.

So is OpenIO made of directories and BLOB storage services:

  • The communication with the directories has to be proxied, this covers the directory of services, and the container services. Here are documented the purpose and interface the interface of the proxy.
  • you get a direct access to the blob storage services

The proxy manages HTTP requests, with a JSON payload. The BLOB requires a subset of WebDAV requests, completed with several expectations on the request headers.

Managing chunks sequences

OpenIO's contents are split into chunks. The collection of chunks is ordered into a sequence, where each items has a position field.

Several chunks might have the same position. They are then considered as replicas from each others. When retrieving the data at this position, the client implementation has the complete freedom of choice.

Compound position are sometimes met, formatted like x.y. The x identifies a specific subset of chunks, then the y each chunk in its sequence. A typical use case is the output of the erasure coding algorithms, when all the chunks under x belong to a same EC compuation.

Upload a content

An upload is currently a 3-step operation:

  • prepare: ask for locations suitable to your container. You ask this to the proxy, and will receive a list of chunks descriptions. It is important to notice that each description contains a position in the chunks sequence. You MUST respect the sequence while uploading the data.
  • push: send the data to each location.
  • commit: tell the container to persist the locations. You need to patch the chunks descriptions with valid chunks hashes, then send this to the proxy.

Prepare

An example prepare is available at OpenIO-Proxyd-API-Reference#prepare-content.

At this point, nothing has been stored yet in the container. If a cancellation is necessary, there is no action to rollback at this point.

Push

Sample RAWX requests have been presented at OpenIO-RAWX-API.

Commit

When the push succeeds, it is time to persist the locations in the container. An example of a final commit is available at OpenIO-Proxyd-API-Reference#create-content.

If too many pushes failed, or the final commit itself, some actions need to be cancelled. It will be necessary to delete the chunks. An example is available at OpenIO-RAWX-API#deleting-a-chunk.

Download a content

Now the upload has been explained, the download should be intuitive:

  • locate: fetch the locations of the chunks of data. You query the proxy for this, and the reply will substancially contain the same chunks descriptions the last upload saved.
  • retrieve: each position has to be downloaded.

When several chunks have the same position, it is up to the client to pick one, and fallback on the others. For the sake of simplicity, you can defer to later the management of compound position.

Fetch the position

The proxy call you need is explained at OpenIO-Proxyd-API-Reference#show-content. It will return to you a content description, in other words a sequene of chunks descriptions completed with several headers related to the content itself. You have now all the information to start downloading yor content's data.

Retrieve the data at each position

Your task is now to issue GET requests on the rawx services. Do you need some tips? They are here: OpenIO-RAWX-API#downloading-a-chunk

List a container

Here, you will only contact the proxy. We only provide a paginated listing feature. All you need is at OpenIO-Proxyd-API-Reference#list-a-container

You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.
Press h to open a hovercard with more details.