Skip to content

[Update] How to Use Linode Object Storage #2609

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jul 29, 2019
Merged

[Update] How to Use Linode Object Storage #2609

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
168 changes: 158 additions & 10 deletions docs/platform/object-storage/how-to-use-object-storage/index.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,6 @@ contributor:
external_resources:
- '[S3cmd usage and command reference](https://s3tools.org/usage)'
---

{{< note >}}
Object Storage is currently in a closed early access Beta, and you may not have access to Object Storage through the Cloud Manager or other tools. To gain access to the Early Access Program (EAP), open up a Customer Support ticket noting that you'd like to be included in the program, or e-mail objbeta@linode.com -- beta access is completely free.

Expand All @@ -26,11 +25,21 @@ Linode's Object Storage is a globally-available, S3- and Swift-compatible method

Additionally, **Object Storage does not require the use of a Linode.** Instead, Object Storage gives each object a unique URL with which you can access your data. An object can be publicly accessible, or you can set it to be private and only visible to you. This makes Object Storage great for sharing and storing unstructured data like images, documents, archives, streaming media assets, and file backups, and the amount of data you store can range from small collections of files up to massive libraries of information. Lastly, Linode Object Storage has the built-in ability to [host a static site](/docs/platform/object-storage/host-static-site-object-storage/).

Below you will find instructions on how to connect to Object Storage, and how to upload and access objects:
In this guide you will learn:

- How to get set-up for object storage by [creating an Object Storage Key Pair](/docs/platform/object-storage/how-to-use-object-storage/#object-storage-key-pair).

- About the variety of first-party and third-party [tools available](/docs/platform/object-storage/how-to-use-object-storage/#object-storage-tools) to access and use the service.

- How to connect to Object Storage, how to upload and access objects, and how to host a static site using:

- [Cloud Manager](/docs/platform/object-storage/how-to-use-object-storage/#cloud-manager)

1. First, you'll need to create a [*key pair*](#object-storage-key-pair) to access the service.
- [Linode CLI](/docs/platform/object-storage/how-to-use-object-storage/#linode-cli)

1. Then, you'll use choose from a variety of available [first-party and third-party tools](#object-storage-tools) to access and use the service.
- [s3cmd](/docs/platform/object-storage/how-to-use-object-storage/#s3cmd)

- [Cyberduck](/docs/platform/object-storage/how-to-use-object-storage/#cyberduck)

## Object Storage Key Pair

Expand Down Expand Up @@ -78,6 +87,8 @@ There are a number of tools that are available to help manage Linode Object Stor

- The [Linode Cloud Manager](#cloud-manager) can be used to create buckets (you are currently not able to upload objects to a bucket from the Cloud Manager).

- The [Linode CLI](#linode-cli) has an Object Storage plugin and can be used to create and remove buckets, add and remove objects, and convert a bucket into a static site from the command line.

- [s3cmd](#s3cmd) is a powerful command line utility that can be used with any S3-compatible object storage service, including Linode's. s3cmd can be used to create and remove buckets, add and remove objects, convert a bucket into a static site from the command line, plus other functions like syncing entire directories up to a bucket.

- [Cyberduck](#cyberduck) is a graphical utility available for Windows and macOS and is a great option if you prefer a GUI tool.
Expand All @@ -98,12 +109,131 @@ The Cloud Manager provides a web interface for creating buckets. To create a buc

![The Create a Bucket menu.](object-storage-create-a-bucket.png)

3. Add a label for your bucket. A bucket's label needs to be unique within the cluster that it lives in, and this includes buckets of the same name on different Linode accounts. If the label you enter is already in use, you will have to choose a different label.
3. Add a label for your bucket.

{{< note >}}
Bucket labels need to be unique within the same cluster, including buckets on other users' Linode accounts. If the label you enter is already in use, you will have to choose a different label. Additionally, bucket labels have the following rules:</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot be formatted as IP addresses</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must be between 3 and 63 characters in length</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Can only contain lower-case characters, numbers, periods, and dashes</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must start with a lowercase letter or number</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot contain underscores (_), end with a dash (-) or period (.), have consecutive periods (.), or use dashes (-) adjacent to periods (.)
{{< /note >}}

4. Choose a cluster location for the bucket to reside in.

5. Click **Submit**. You are now ready to upload objects to your bucket using one of the other tools outlined in this guide.

## Linode CLI

The Linode Command Line Interface (CLI) is a command line utility that allows you complete control over your Linode account. With the Object Storage plugin, you can also create and remove buckets, upload objects, and more.

### Install and Configure the CLI

1. Download the Linode CLI, or, if you have already downloaded it, make sure it has been upgraded to the latest version:

pip install linode-cli --upgrade

1. Configure the Object Storage plugin:

linode-cli obj --help

You will be prompted to enter in your Personal Access Token and default settings for deploying new Linodes.

1. Install the `boto` module:

pip install boto

Now you are ready to create buckets and upload objects.

### Create a Bucket with the CLI

To create a bucket with the Linode CLI, issue the `mb` command.

linode-cli obj mb my-example-bucket

{{< note >}}
Bucket labels need to be unique within the same cluster, including buckets on other users' Linode accounts. If the label you enter is already in use, you will have to choose a different label. Additionally, bucket labels have the following rules:</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot be formatted as IP addresses</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must be between 3 and 63 characters in length</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Can only contain lower-case characters, numbers, periods, and dashes</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must start with a lowercase letter or number</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot contain underscores (_), end with a dash (-) or period (.), have consecutive periods (.), or use dashes (-) adjacent to periods (.)
{{< /note >}}

To delete a bucket, issue the `rb` command:

linode-cli obj rb my-example-bucket

If your bucket has objects in it, you will not be able to immediately delete it from the Linode CLI. Instead, remove the objects first, then delete the bucket. The [s3cmd](/docs/platform/object-storage/how-to-use-object-storage/#s3cmd) tool has commands for deleting all objects from a bucket, and it can also force-delete a bucket with objects in it.

### Upload, Download, and Delete an Object with the CLI

1. As an example object, create a text file and fill it with some example text.

echo 'Hello World!' > example.txt

1. To upload an object to a bucket using the Linode CLI, issue the `put` command. Supply the object name as the first parameter and the bucket label as the second:

linode-cli obj put --acl-public example.txt my-example-bucket

The file will now be accessible at the URL `http://my-example-bucket.us-east-1.linodeobjects.com/example.txt`.

{{< note >}}
The `--acl-public` flag is used to make the object publicly accessible, meaning that you will be able to access the object from its URL. By default, all objects are set to private. To make a public file private, or a private file public, use the `setacl` command and supply the corresponding flag.

For instance, if you want to make a public file private, you would supply the `--acl-private` flag:

linode-cli obj setacl --acl-private my-example-bucket example.txt
{{</ note >}}

1. To download an object, issue the `get` command. Supply the label of the bucket as the first parameter and the name of the file as the second:

linode-cli obj get my-example-bucket example.txt

1. To delete an object, issue the `rm` or `del` command. Supply the label of the bucket as the first parameter and the name of the object as the second:

linode-cli obj rm my-example-bucket example.txt

### Create a Static Site with the CLI

To create a static website from a bucket:

1. Issue the `ws-create` command, including the `--ws-index` and `--ws-error` flags:

linode-cli obj ws-create my-example-bucket --ws-index=index.html --ws-error=404.html

The `--ws-index` and `--ws-error` flags specify which objects the bucket should use to serve the static site's index page and error page, respectively.

1. You need to separately upload the `index.html` and `404.html` files (or however you have named the index and error pages) to your bucket:

echo 'Index page' > index.html
echo 'Error page' > 404.html
linode-cli obj put index.html 404.html my-example-bucket

1. Set the `--aclpublic` flag on both the `index.html` and `404.html` files:

linode-cli obj setacl --acl-public my-example-bucket index.html
linode-cli obj setacl --acl-public my-example-bucket 404.html

1. Your static site is accessed from a different URL than the generic URL for your Object Storage bucket. Static sites are available at the `website-us-east-1` subdomain. Using `my-example-bucket` as an example, navigate to `http://my-example-bucket.website-us-east-1.linodeobjects.com`.

For more information on hosting static websites from Linode Object Storage, see our [Host a Static Site on Linode's Object Storage](/docs/platform/object-storage/host-static-site-object-storage/) guide.

### Other CLI Commands

To get a list of all available buckets, issue the `ls` command:

linode-cli obj ls

To get a list of all objects in a bucket, issue the `ls` command with the label of a bucket:

linode-cli obj ls my-example-bucket

For a complete list of commands available with the Object Storage plugin, use the `--help` flag:

linode-cli obj --help

## s3cmd

s3cmd is a command line utility that you can use for any S3-compatible Object Storage.
Expand Down Expand Up @@ -165,10 +295,19 @@ You are now ready to use s3cmd to create a bucket in Object Storage.

### Create a Bucket with s3cmd

You can create a bucket with s3cmd issuing the following `mb` command, replacing `my-example-bucket` with the name of the bucket you would like to create. Bucket names need to be unique within the same cluster, including buckets on other Linode accounts. If you choose a name for your bucket that someone else has already created, you will have to choose a different name:
You can create a bucket with s3cmd issuing the following `mb` command, replacing `my-example-bucket` with the label of the bucket you would like to create.

s3cmd mb s3://my-example-bucket

{{< note >}}
Bucket labels need to be unique within the same cluster, including buckets on other users' Linode accounts. If the label you enter is already in use, you will have to choose a different label. Additionally, bucket labels have the following rules:</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot be formatted as IP addresses</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must be between 3 and 63 characters in length</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Can only contain lower-case characters, numbers, periods, and dashes</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must start with a lowercase letter or number</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot contain underscores (_), end with a dash (-) or period (.), have consecutive periods (.), or use dashes (-) adjacent to periods (.)
{{< /note >}}

To remove a bucket, you can use the `rb` command:

s3cmd rb s3://my-example-bucket
Expand Down Expand Up @@ -206,7 +345,7 @@ If you chose to enable encryption when configuring s3cmd, you can store encrypte
Public URL of the object is: http://us-east-1.linodeobjects.com/my-example-bucket/example.txt

{{< note >}}
The URL for the object that s3cmd provides is one of two valid ways to access your object. The first, which s3cmd provides, places the name of your bucket after the domain name. You can also access your object by affixing your bucket name as a subdomain: `http://my-example-bucket.us-east-1.linodeobjects.com/example.txt`. The latter URL is generally favored.
The URL for the object that s3cmd provides is one of two valid ways to access your object. The first, which s3cmd provides, places the label of your bucket after the domain name. You can also access your object by affixing your bucket label as a subdomain: `http://my-example-bucket.us-east-1.linodeobjects.com/example.txt`. The latter URL is generally favored.
{{< /note >}}

1. To retrieve a file, issue the `get` command:
Expand Down Expand Up @@ -297,7 +436,16 @@ To create a bucket in Cyberduck:

![Right click or click 'Action', then click 'New Folder'](object-storage-cyberduck-create-bucket.png)

1. Enter your bucket's name and then click **Create**. Bucket names need to be unique within the same cluster, including buckets on other Linode accounts. If the name of your bucket is already in use, you will have to choose a different name.
1. Enter your bucket's label and then click **Create**.

{{< note >}}
Bucket labels need to be unique within the same cluster, including buckets on other users' Linode accounts. If the label you enter is already in use, you will have to choose a different label. Additionally, bucket labels have the following rules:</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot be formatted as IP addresses</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must be between 3 and 63 characters in length</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Can only contain lower-case characters, numbers, periods, and dashes</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Must start with a lowercase letter or number</br>
&nbsp;&nbsp;&nbsp;&bull;&nbsp;&nbsp;Cannot contain underscores (_), end with a dash (-) or period (.), have consecutive periods (.), or use dashes (-) adjacent to periods (.)
{{< /note >}}

To delete the bucket using Cyberduck, right click on the bucket and select **Delete**.

Expand All @@ -319,7 +467,7 @@ To delete the bucket using Cyberduck, right click on the bucket and select **Del

![Set the permissions for 'Everyone' to READ.](object-storage-cyberduck-object-permissions2.png)

Your object is now accessible via the internet, at the URL `http://my-example-bucket.us-east-1.linodeobjects.com/example.txt`, where `my-example-bucket` is the name of your bucket, and `example.txt` is the name of your object.
Your object is now accessible via the internet, at the URL `http://my-example-bucket.us-east-1.linodeobjects.com/example.txt`, where `my-example-bucket` is the label of your bucket, and `example.txt` is the name of your object.

1. To download an object, right click on the object and select **Download**, or click **Download As** if you'd like to specify the location of the download.

Expand All @@ -345,4 +493,4 @@ To create a static site from your bucket:

## Next Steps

There are S3 bindings available for a number of programming languages, including the popular [Boto](https://github.com/boto/boto3) library for Python, that allow you to interact with Object Storage programmatically.
There are S3 bindings available for a number of programming languages, including the popular [Boto](https://github.com/boto/boto3) library for Python, that allow you to interact with Object Storage programmatically.