Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Write your own client
Ok, you are about to write your own OpenIO SDS client. This is nice, welcome in our community !
- General design
- Managing chunks sequences
- Upload a content
- Download a content
- List a container
First, you should consider reading OpenIO-Architecture. This is not mandatory but it will give you the keys to understand our choices.
Using OpenIO means handling objets referenced in container. You should at least be comfortable with our naming conventions, described at OpenIO-Object-Names.
Working with a client API brings your application into your OpenIO grid, while working with a gateway keeps the applicaiton aside of the grid. You need to know what you are doing, because you can do things really wrong.
Eventually you will want to test your code, then OpenIO-Local-Sandbox contains precious information that will help you setting up a local development environment.
OpenIO has been initially designed for emails platforms: a lot of small files, organized into a lot of containers. The entities you will have to manage reflect this design.
So is OpenIO made of directories and BLOB storage services:
- The communication with the directories has to be proxied, this covers the directory of services, and the container services. Here are documented the purpose and interface the interface of the proxy.
- you get a direct access to the blob storage services
The proxy manages HTTP requests, with a JSON payload. The BLOB requires a subset of WebDAV requests, completed with several expectations on the request headers.
Managing chunks sequences
OpenIO's contents are split into chunks.
The collection of chunks is ordered into a sequence, where each items has a
Several chunks might have the same position. They are then considered as replicas from each others. When retrieving the data at this position, the client implementation has the complete freedom of choice.
Compound position are sometimes met, formatted like
x identifies a specific subset of chunks, then the
y each chunk in its sequence.
A typical use case is the output of the erasure coding algorithms, when all the chunks under
x belong to a same EC compuation.
Upload a content
An upload is currently a 3-step operation:
- prepare: ask for locations suitable to your container. You ask this to the proxy, and will receive a list of chunks descriptions. It is important to notice that each description contains a position in the chunks sequence. You MUST respect the sequence while uploading the data.
- push: send the data to each location.
- commit: tell the container to persist the locations. You need to patch the chunks descriptions with valid chunks hashes, then send this to the proxy.
An example prepare is available at OpenIO-Proxyd-API-Reference#prepare-content.
At this point, nothing has been stored yet in the container. If a cancellation is necessary, there is no action to rollback at this point.
Sample RAWX requests have been presented at OpenIO-RAWX-API.
When the push succeeds, it is time to persist the locations in the container. An example of a final commit is available at OpenIO-Proxyd-API-Reference#create-content.
If too many pushes failed, or the final commit itself, some actions need to be cancelled. It will be necessary to delete the chunks. An example is available at OpenIO-RAWX-API#deleting-a-chunk.
Download a content
Now the upload has been explained, the download should be intuitive:
- locate: fetch the locations of the chunks of data. You query the proxy for this, and the reply will substancially contain the same chunks descriptions the last upload saved.
- retrieve: each position has to be downloaded.
When several chunks have the same position, it is up to the client to pick one, and fallback on the others. For the sake of simplicity, you can defer to later the management of compound position.
Fetch the position
The proxy call you need is explained at OpenIO-Proxyd-API-Reference#show-content. It will return to you a content description, in other words a sequene of chunks descriptions completed with several headers related to the content itself. You have now all the information to start downloading yor content's data.
Retrieve the data at each position
Your task is now to issue
GET requests on the
Do you need some tips? They are here: OpenIO-RAWX-API#downloading-a-chunk
List a container
Here, you will only contact the
We only provide a paginated listing feature.
All you need is at OpenIO-Proxyd-API-Reference#list-a-container