Skip to content
This repository has been archived by the owner on Jul 6, 2023. It is now read-only.

Latest commit

 

History

History
685 lines (604 loc) · 26.4 KB

api.md

File metadata and controls

685 lines (604 loc) · 26.4 KB
Raw

Contents

Overview

Heketi provides a RESTful management interface which can be used to manage the life cycle of GlusterFS volumes. The goal of Heketi is to provide a simple way to create, list, and delete GlusterFS volumes in multiple storage clusters. Heketi intelligently will manage the allocation, creation, and deletion of bricks throughout the disks in the cluster. Heketi first needs to learn about the topologies of the clusters before satisfying any requests. It organizes data resources into the following: Clusters, contain Nodes, which contain Devices, which will contain Bricks.

Development

To communicate with the Heketi service, you will need to either use a client library or directly communicate with the REST endpoints. The following client libraries are supported: Go, Python

Go Client Library

Here is a small example of how to use the Go client library:

import (
	"fmt"
	"github.com/heketi/heketi/client/api/go-client"
)

func main() {
	// Create a client
	heketi := client.NewClient(options.Url, options.User, options.Key)

	// List clusters
	list, err := heketi.ClusterList()
	if err != nil {
		return err
	}

	output := strings.Join(list.Clusters, "\n")
	fmt.Fprintf(stdout, "Clusters:\n%v\n", output)
	return nil
}

For more examples see the Heketi cli client.

Python Client Library

The python client library can be installed either from the source or by installing the python-heketi package in Fedora/RHEL. The source is available https://github.com/heketi/heketi/tree/master/client/api/python , and for examples, please check out the unit tests

Running the development server

The simplest way to development a client for Heketi is to run the Heketi service in mock mode. In this mode, Heketi will not need to communicate with any storage nodes, instead it mocks the communication, while still supporting all REST calls and maintaining state. The simplest way to run the Heketi server is to run it from a container as follows:

# docker run -d -p 8080:8080 heketi/heketi
# curl http://localhost:8080/hello
Hello from Heketi

Authentication Model

Heketi uses a stateless authentication model based on the JSON Web Token (JWT) standard as proposed to the IETF. As specified by the specification, a JWT token has a set of claims which can be added to a token to determine its correctness. Heketi requires the use of the following standard claims:

  • iss: Issuer. Heketi supports two types of issuers:
    • admin: Has access to all APIs
    • user: Has access to only Volume APIs
  • iat: Issued-at-time
  • exp: Time when the token should expire

And a custom one following the model as described on Atlassian:

  • qsh. URL Tampering prevention.

Heketi supports token signatures encrypted using the HMAC SHA-256 algorithm which is specified by the specification as HS256.

Clients

There are JWT libraries available for most languages as highlighted on jwt.io. The client libraries allow you to easily create a JWT token which must be stored in the Authorization: Bearer {token} header. A new token will need to be created for each REST call. Here is an example of the header:

Authorization: Bearer eyJhb[...omitted for brevity...]HgQ

More Information

Asynchronous Operations

Some operations may take a long time to process. For these operations, Heketi will return 202 Accepted with a temporary resource set inside the Location header. A client can then issue a GET on this temporary resource and receive the following:

  • HTTP Status 200: Request is still in progress. We may decide to add some JSON ETA data here in future releases.
    • Header X-Pending will be set to the value of true
  • HTTP Status 404: Temporary resource requested is not found.
  • HTTP Status 500: Request completed and has failed. Body will be filled in with error information.
  • HTTP Status 303 See Other: Request has been completed successfully. The information requested can be retrieved by issuing a GET on the resource set inside the Location header.
  • HTTP Status 204 Done: Request has been completed successfully. There is no data to return.

API

Heketi uses JSON as its data serialization format. XML is not supported.

Most APIs use use the following methods on the URIs:

  • URIs in the form of /<REST endpoint>/{id}
  • Responses and requests are in JSON format
  • POST: Send data to Heketi where the body has data described in JSON format.
  • GET: Retrieve data from Heketi where the body has data described in JSON format.
  • DELETE: Deletes the specified object from Heketi.

Before servicing any requests, Heketi must first learn the topology of the clusters. Once it knows which nodes and disks to use, it can then service requests.

Clusters

Heketi is able to manage multiple GlusterFS clusters, each composed of a set of storage nodes. Once a cluster has been created, nodes can then be added to it for Heketi to manage. A GlusterFS cluster is a set of nodes participating as a trusted storage pool. Volumes do not cross cluster boundaries.

Create Cluster

  • Method: POST
  • Endpoint:/clusters
  • Content-Type: application/json
  • Response HTTP Status Code: 201
  • JSON Request: Empty body, or a JSON request with optional attributes:
    • file: bool, optional, whether this cluster should allow creation of file volumes (default: true)
    • block: bool, optional, whether this cluster should allow creation of block volumes (default: true)
    • Example:
{
"block" : false
}
{
    "id": "67e267ea403dfcdf80731165b300d1ca",
    "nodes": [],
    "volumes": [],
}

Set Cluster Flags

  • Method: POST
  • Endpoint:/clusters/{id}/flags
  • Content-Type: application/json
  • Response HTTP Status Code: 200
  • JSON Request:
    • file: bool, whether this cluster should allow creation of file volumes
    • block: bool, whether this cluster should allow creation of block volumes
    • Example:
{
    "file": true,
    "block": false,
}
  • JSON Response: None

Cluster Information

  • Method: GET
  • Endpoint:/clusters/{id}
  • Response HTTP Status Code: 200
  • JSON Request: None
  • JSON Response:
    • id: string, UUID for node
    • nodes: array of strings, UUIDs of each node in the cluster
    • volumes: array of strings, UUIDs of each volume in the cluster
    • Example:
{
    "id": "67e267ea403dfcdf80731165b300d1ca",
    "nodes": [
        "78696abbba372659effa",
        "799029acaa867a66934"
    ],
    "volumes": [
        "aa927734601288237463aa",
        "70927734601288237463aa"
    ],
}

List Clusters

  • Method: GET
  • Endpoint:/clusters
  • Response HTTP Status Code: 200
  • JSON Request: None
  • JSON Response:
    • clusters: array of strings, UUIDs of clusters
    • Example:
{
    "clusters": [
        "67e267ea403dfcdf80731165b300d1ca",
        "ff6667ea403dfcdf80731165b300d1ca"
    ]
}

Delete Cluster

  • Method: DELETE
  • Endpoint:/clusters/{id}
  • Response HTTP Status Code: 200
  • Response HTTP Status Code: 409, Returned if it contains nodes
  • JSON Request: None
  • JSON Response: None

Nodes

The node RESTful endpoint is used to register a storage system for Heketi to manage. Devices in this node can then be registered.

Add Node

  • Method: POST
  • Endpoint:/nodes
  • Content-Type: application/json
  • Response HTTP Status Code: 202, See Asynchronous Operations
  • Temporary Resource Response HTTP Status Code: 303, Location header will contain /nodes/{id}. See Node Info for JSON response.
  • JSON Request:
    • zone: int, failure domain. Value of 0 is not allowed.
    • hostnames: map of strings
      • manage: array of strings, List of node management hostnames.
        • In SSH configurations, Heketi needs to be able to SSH to the host on any of the supplied management hostnames. It is highly recommended to use hostnames instead of IP addresses.
        • For Kubernetes and OpenShift, this must be the name of the node as shown in kubectl get nodes which is running the Gluster POD
        • NOTE: Even though it takes a list of hostnames, only one is supported at the moment. The plan is to support multiple hostnames when glusterd-2 is used. For Kubernetes and OpenShift, this must be the name of the Pod file, not the name of the node.
      • storage: array of strings, List of node storage network hostnames. These storage network addresses will be used to create and access the volume. It is highly recommended to use hostnames instead of IP addresses. NOTE: Even though it takes a list of hostnames, only one is supported at the moment. The plan is to support multiple ip address when glusterd-2 is used.
    • cluster: string, UUID of cluster to whom this node should be part of.
    • tags: map of strings, (optional) a mapping of tag-names to tag-values
    • Example:
{
    "zone": 1,
    "hostnames": {
        "manage": [
            "node1-manage.gluster.lab.com"
        ],
        "storage": [
            "node1-storage.gluster.lab.com"
        ]
    },
    "tags": {
        "incantation": "abracadabra"
    },
    "cluster": "67e267ea403dfcdf80731165b300d1ca"
}

Node Information

  • Method: GET
  • Endpoint:/nodes/{id}
  • Response HTTP Status Code: 200
  • JSON Request: None
  • JSON Response:
    • zone: int, Failure Domain
    • id: string, UUID for node
    • cluster: string, UUID of cluster
    • hostnames: map of strings
      • manage: array of strings, List of node management hostnames. Heketi needs to be able to SSH to the host on any of the supplied management hostnames.
      • storage: array of strings, List of node storage network hostnames. These storage network addresses will be used to create and access the volume.
    • devices: array maps, See Device Information
    • tags: map, (omitted if empty) a mapping of tag-names to tag-values
    • Example:
{
    "zone": 1,
    "id": "88ddb76ad403dfcdf80731165b300d1ca",
    "cluster": "67e267ea403dfcdf80731165b300d1ca",
    "hostnames": {
        "manage": [
            "node1-manage.gluster.lab.com"
        ],
        "storage": [
            "node1-storage.gluster.lab.com"
        ]
    },
    "tags": {
        "arbiter": "supported",
        "rack": "7,4"
    },
    "devices": [
        {
            "name": "/dev/sdh",
            "storage": {
                "total": 2000000,
                "free": 2000000,
                "used": 0
            },
            "id": "49a9bd2e40df882180479024ac4c24c8",
            "bricks": [
                {
                    "id": "aaaaaad2e40df882180479024ac4c24c8",
                    "path": "/gluster/brick_aaaaaad2e40df882180479024ac4c24c8/brick",
                    "size": 0
                },
                {
                    "id": "bbbbbbd2e40df882180479024ac4c24c8",
                    "path": "/gluster/brick_bbbbbbd2e40df882180479024ac4c24c8/brick",
                    "size": 0
                }
            ]
        }
    ]
}

Set Node Tags

Allows setting, updating, and deleting user specified metadata tags on a node. Tags are key-value pairs that are stored on the server. Certain well-known tags may be used by the server for configuration.

Specifying a change_type of set overwrites the tags on the object with exactly the set of tags in this request. A change_type of update adds new tags and updates existing tags without changing tags not specified in the request. A change_type of delete will remove tags names in the request (tag values in the request will be ignored).

  • Method: POST
  • Endpoint: /nodes/{id}/tags
  • Response HTTP Status Code: 200
  • JSON Request: None
    • change_type: string, one of "set", "update", "delete"
    • tags: map of strings, a mapping of tag-names to tag-values
    • Example:
{
    "change_type": "set",
    "tags": {
        "arbiter": "supported",
        "rack": "7,4",
        "os_version": "linux 4.15.8"
    }
}
  • JSON Response: Ignored

Delete Node

  • Method: DELETE
  • Endpoint:/nodes/{id}
  • Response HTTP Status Code: 202, See Asynchronous Operations
  • Response HTTP Status Code: 409, Node contains devices
  • Temporary Resource Response HTTP Status Code: 204

Devices

The devices endpoint allows management of raw devices in the cluster.

Add Device

  • Method: POST
  • Endpoint:/devices
  • Content-Type: application/json
  • Response HTTP Status Code: 202, See Asynchronous Operations
  • Temporary Resource Response HTTP Status Code: 204
  • JSON Request:
    • node: string, UUID of node which the devices belong to.
    • name: string, Device name
    • destroydata: bool, (optional) destroy any data on the device
    • tags: map of strings, (optional) a mapping of tag-names to tag-values
    • Example:
{
    "node": "714c510140c20e808002f2b074bc0c50",
    "name": "/dev/sdb",
    "tags": {
        "serial_number": "a109-84338af8e43-48dd9d43-919231"
    }
}

Device Information

  • Method: GET
  • Endpoint:/devices/{id}
  • Response HTTP Status Code: 200
  • JSON Request: None
  • JSON Response:
    • name: string, Name of device
    • id: string, UUID of device
    • total: uint64, Total storage in KB
    • free: uint64, Available storage in KB
    • used: uint64, Allocated storage in KB
    • bricks: array of maps, Bricks allocated on this device
      • id: string, UUID of brick
      • path: string, Path of brick on the node
      • size: uint64, Size of brick in KB
    • tags: map, (omitted if empty) a mapping of tag-names to tag-values
    • Example:
{
    "name": "/dev/sdh",
    "storage": {
        "total": 1000000,
        "free": 1000000,
        "used": 0
    },
    "id": "49a9bd2e40df882180479024ac4c24c8",
    "tags": {
        "arbiter": "required",
        "drivebay": "3"
    },
    "bricks": [
        {
            "id": "aaaaaad2e40df882180479024ac4c24c8",
            "path": "/gluster/brick_aaaaaad2e40df882180479024ac4c24c8/brick",
            "size": 0,
            "node": "714c510140c20e808002f2b074bc0c50",
            "device": "49a9bd2e40df882180479024ac4c24c8"
        },
        {
            "id": "bbbbbbd2e40df882180479024ac4c24c8",
            "path": "/gluster/brick_bbbbbbd2e40df882180479024ac4c24c8/brick",
            "size": 0,
            "node": "714c510140c20e808002f2b074bc0c50",
            "device": "49a9bd2e40df882180479024ac4c24c8"
        }
    ]
}

Set Device Tags

Allows setting, updating, and deleting user specified metadata tags on a device. Tags are key-value pairs that are stored on the server. Certain well-known tags may be used by the server for configuration.

Specifying a change_type of set overwrites the tags on the object with exactly the set of tags in this request. A change_type of update adds new tags and updates existing tags without changing tags not specified in the request. A change_type of delete will remove tags names in the request (tag values in the request will be ignored).

  • Method: POST
  • Endpoint: /devices/{id}/tags
  • Response HTTP Status Code: 200
  • JSON Request: None
    • change_type: string, one of "set", "update", "delete"
    • tags: map of strings, a mapping of tag-names to tag-values
    • Example:
{
    "change_type": "set",
    "tags": {
        "arbiter": "required",
        "drivebay": "3",
        "serial_number": "a109-84338af8e43-48dd9d43-919231"
    }
}
  • JSON Response: Ignored

Delete Device

  • Method: DELETE
  • Endpoint:/devices/{id}
  • Response HTTP Status Code: 202, See Asynchronous Operations
  • Response HTTP Status Code: 409, Device contains bricks
  • Temporary Resource Response HTTP Status Code: 204

Volumes

These APIs inform Heketi to create a network file system of a certain size available to be used by clients.

Create a Volume

  • Method: POST
  • Endpoint:/volumes
  • Content-Type: application/json
  • Response HTTP Status Code: 202, See Asynchronous Operations
  • Temporary Resource Response HTTP Status Code: 303, Location header will contain /volumes/{id}. See Volume Info for JSON response.
  • JSON Request:
    • size: int, Size of volume requested in GiB
    • name: string, optional, Name of volume. If not provided, the name of the volume will be vol_{id}, for example vol_728faa5522838746abce2980
    • durability: map, optional, Durability Settings
      • type: string, optional, Durability type. Choices are none (Distributed Only), replicate (Distributed-Replicated), disperse (Distributed-Disperse). If omitted, durability type will default to none.
      • replicate: map, optional, Replica settings, only used if type is set to replicate.
        • replica: int, optional, Number of replica per brick. If omitted, it will default to 2.
      • disperse: map, optional, Erasure Code settings, only used if type is set to disperse.
        • data: int, optional Number of dispersed data volumes. If omitted, it will default to 8.
        • redundancy: int, optional, Level of redundancy. If omitted, it will default to 2.
    • snapshot: map
      • enable: bool, optional, Snapshot support requested for this volume. If omitted, it will default to false.
      • factor: float32, optional, Snapshot reserved space factor. When creating a volume with snapshot enabled, the size of the brick will be set to factor * brickSize, where brickSize is automatically determined to satisfy the volume size request. If omitted, it will default to 1.5.
        • Requirement: Value must be greater than one.
    • clusters: array of string, optional, UUIDs of clusters where the volume should be created. If omitted, each cluster will be checked until one is found that can satisfy the request.
    • Example:
{
    "size": 10000000,
    "snapshot": {
        "enable": true,
        "factor": 1.2
    },
    "durability": {
        "type": "replicate",
        "replicate": {
            "replica": 3
        }
    },
    "clusters": [
        "2f84c71240f43e16808bc64b05ad0d06",
        "5a2c52d04075373e80dbfa1e291ba0de"
    ]
}

Note: The volume size created depends upon the underlying brick size. For example, for a 2 way/3 way replica volume, the minimum volume size is 1GiB as the underlying minimum brick size is constrained to 1GiB.

So, it is not possible create a volume of size less than 1GiB.

Volume Information

  • Method: GET
  • Endpoint:/volumes/{id}
  • Response HTTP Status Code: 200
  • JSON Request: None
  • JSON Response:
    • name: string, Name of volume
    • size: int, Size of volume in GiB
    • id: string, Volume UUID
    • cluster: string, UUID of cluster which contains this volume
    • durability: map, Durability settings. See Volume Create for more information.
    • snapshot: map, If omitted, snapshots are disabled.
      • enable: bool, Snapshot support requested for this volume.
      • factor: float32, optional, Snapshot reserved space factor if enabled
    • replica: int, Replica count
    • mounts: map, Information used to mount or gain access to the network volume file system
      • glusterfs: map, Mount point information for native GlusterFS FUSE mount
        • device: string, Mount point used for native GlusterFS FUSE mount
        • options: map, Optional mount options to use
          • backup-volfile-servers: string, List of backup volfile servers [1] [2] [3]. It is up to the calling service to determine which of the volfile servers to use in the actual mount command.
    • brick: array of maps, Bricks used to create volume. See Device Information for brick JSON description
    • Example:
{
    "name": "vol_70927734601288237463aa",
    "id": "70927734601288237463aa",
    "cluster": "67e267ea403dfcdf80731165b300d1ca",
    "size": 123456,
    "durability": {
        "type": "replicate",
        "replicate": {
            "replica": 3
        }
    },
    "snapshot": {
        "enable": true,
        "factor": 1.2
    },
    "mount": {
        "glusterfs": {
            "device": "192.168.1.103:vol_70927734601288237463aa",
            "options": {
                "backup-volfile-servers": "192.168.1.103,192.168.101"
            }
        }
    },
    "bricks": [
        {
            "id": "aaaaaad2e40df882180479024ac4c24c8",
            "path": "/gluster/brick_aaaaaad2e40df882180479024ac4c24c8/brick",
            "size": 0,
            "node": "892761012093474071983852",
            "device": "ff2137326add231578ffa7234"
        },
        {
            "id": "bbbbbbd2e40df882180479024ac4c24c8",
            "path": "/gluster/brick_bbbbbbd2e40df882180479024ac4c24c8/brick",
            "size": 0,
            "node": "714c510140c20e808002f2b074bc0c50",
            "device": "49a9bd2e40df882180479024ac4c24c8"
        }
    ]
}

Expand a Volume

New volume size will be reflected in the volume information.

  • Method: POST
  • Endpoint:/volumes/{id}/expand
  • Content-Type: application/json
  • Response HTTP Status Code: 202, See Asynchronous Operations
  • Temporary Resource Response HTTP Status Code: 303, Location header will contain /volumes/{id}. See Volume Info for JSON response.
  • JSON Request:
    • expand_size: int, Amount of storage to add to the existing volume in GiB
{ "expand_size" : 1000000 }

Delete Volume

When a volume is deleted, Heketi will first stop, then destroy the volume. Once destroyed, it will remove the allocated bricks and free the allocated space.

  • Method: DELETE
  • Endpoint:/volumes/{id}
  • Response HTTP Status Code: 202, See Asynchronous Operations
  • Temporary Resource Response HTTP Status Code: 204

List Volumes

  • Method: GET
  • Endpoint:/volumes
  • Response HTTP Status Code: 200
  • JSON Response:
    • volumes: array strings, List of volume UUIDs.
    • Example:
{
    "volumes": [
        "aa927734601288237463aa",
        "70927734601288237463aa"
    ]
}

Get Metrics

Get current metrics for the heketi cluster. Metrics are exposed in the prometheus format.

  • Method: GET
  • Endpoint:/metrics
  • Response HTTP Status Code: 200
  • TEXT Response:
    • Example:
# HELP heketi_cluster_count Number of clusters
# TYPE heketi_cluster_count gauge
heketi_cluster_count 1
# HELP heketi_device_brick_count Number of bricks on device
# TYPE heketi_device_brick_count gauge
heketi_device_brick_count{cluster="c1",device="d1",hostname="n1"} 1
# HELP heketi_device_count Number of devices on host
# TYPE heketi_device_count gauge
heketi_device_count{cluster="c1",hostname="n1"} 1
# HELP heketi_device_free Amount of Free space available on the device
# TYPE heketi_device_free gauge
heketi_device_free{cluster="c1",device="d1",hostname="n1"} 1
# HELP heketi_device_size Total size of the device
# TYPE heketi_device_size gauge
heketi_device_size{cluster="c1",device="d1",hostname="n1"} 2
# HELP heketi_device_used Amount of space used on the device
# TYPE heketi_device_used gauge
heketi_device_used{cluster="c1",device="d1",hostname="n1"} 1
# HELP heketi_nodes_count Number of nodes on cluster
# TYPE heketi_nodes_count gauge
heketi_nodes_count{cluster="c1"} 1
# HELP heketi_up Is heketi running?
# TYPE heketi_up gauge
heketi_up 1
# HELP heketi_volumes_count Number of volumes on cluster
# TYPE heketi_volumes_count gauge
heketi_volumes_count{cluster="c1"} 0