Skip to content
master
Switch branches/tags
Code

Latest commit

 

Git stats

Files

Permalink
Failed to load latest commit information.
Type
Name
Latest commit message
Commit time
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

HSDS (Highly Scalable Data Service) - REST-based service for HDF5 data

Introduction

HSDS is a web service that implements a REST-based web service for HDF5 data stores. Data can be stored in either a POSIX files system, or using object based storage such as AWS S3, Azure Blob Storage, or OpenIO <openio.io>. HSDS can be run a single machine using Docker or on a cluster using Kubernetes (or AKS on Microsoft Azure).

Websites

Other useful resources

Quick Start

Make sure you have Python 3, docker, docker-compose installed, then:

  1. Clone this repo: $ git clone https://github.com/HDFGroup/hsds
  2. Go to the hsds directory: $ cd hsds
  3. Run install: $ python setup.py install
  4. Setup password file: $ cp admin/config/passwd.default admin/config/passwd.txt
  5. Create a directory the server will use to store data, and then set the ROOT_DIR environment variable to point to it: $ mkdir ~/hsds_data; export ROOT_DIR="${HOME}/hsds_data"
  6. Start server: $ ./runall.sh
  7. Try making a request to the service: $ curl http://localhost:5101/about (should get back a json response)
  8. Set environment variables for the admin password and username: $ export ADMIN_PASSWORD=admin, $ export ADMIN_USERNAME=admin
  9. Run the test suite: $ python testall.py --skip_unit
  10. (Optional) Post install setup (test data, home folders, cli tools, etc): docs/post_install.md

To shut down the server, run: $ ./stopall.sh

Note: passwords can (and should for production use) be modified by changing values in hsds/admin/config/password.txt and rebuilding the docker image. Alternatively, an external identity provider such as Azure Active Directory or KeyCloak can be used. See: docs/azure_ad_setup.md for Azure AD setup instructions or docs/keycloak_setup.md for KeyCloak.

Detailed Install Instructions

On AWS

See: docs/docker_install_aws.md for complete install instructions.

See: docs/kubernetes_install_aws.md for setup on Kubernetes.

On Azure

For complete instructions to install on a single Azure VM:

For complete instructions to install on Azure Kubernetes Service (AKS):

On Prem (POSIX-based storage)

For complete instructions to install on a desktop or local server:

On DCOS (BETA)

For complete instructions to install on DCOS:

General Install Topics

Setting up docker:

Post install setup and testing:

Authorization, ACLs, and Role Based Access Control (RBAC):

Writing Client Applications

As a REST service, clients be developed using almost any programming language. The test programs under: hsds/test/integ illustrate some of the methods for performing different operations using Python and HSDS REST API (using the requests package).

The related project: https://github.com/HDFGroup/h5pyd provides a (mostly) h5py-compatible interface to the server for Python clients.

For C/C++ clients, the HDF REST VOL is a HDF5 library plugin that enables the HDF5 API to read and write data using HSDS. See: https://github.com/HDFGroup/vol-rest. Note: requires v1.12.0 or greater version of the HDF5 library.

Uninstalling

HSDS only modifies the storage location that it is configured to use, so to uninstall just remove source files, Docker images, and S3 bucket/Azure Container/directory files.

Reporting bugs (and general feedback)

Create new issues at http://github.com/HDFGroup/hsds/issues for any problems you find.

For general questions/feedback, please use the HSDS forum: https://forum.hdfgroup.org/c/hsds.

License

HSDS is licensed under an APACHE 2.0 license. See LICENSE in this directory.

Integration with JupyterHub

The HDF Group provides access to an HSDS instance that is integrated with JupyterLab: HDF Lab. HDF Lab is a hosted Jupyter environment with these features:

  • Connection to a HSDS instance
  • Dedicated Xeon core per user
  • 10 GB Posix Disk
  • 200 GB S3 storage for HDF data
  • Sample programs and data files

Sign up for HDF Lab here: https://www.hdfgroup.org/hdfkitalab/.

Azure Marketplace

VM Offer for Azure Marketplace. HSDS for Azure Marketplace provides an easy way to setup a Azure instance with HSDS. See: https://azuremarketplace.microsoft.com/en-us/marketplace/apps/thehdfgroup1616725197741.hsdsazurevm?tab=Overview for more information.