This repository contains a few scenarios for quickly and easily getting started with Onedata distributed data management platform. The scenarios are intended for administrators who want to get familiar with Onedata.
If you're new to Onedata please have a look at the introductory information in our documentation.
Scenarios vary in complexity: beginning with simple, preconfigured demos and ending with advanced multi-cluster setups. The scenarios are implemented using preconfigured Docker images, and can be quickly deployed and tested.
We plan to add more scenarios in the future, if you can't find a scenario that addresses your specific requirements please create an issue. If you experience any problems or have questions you welcome to visit our support channel.
Scenarios are divided into 2 groups:
- 1.0: pre-configured Oneprovider that connects to beta.onedata.org zone (public IP required) <br > Sources: scenarios/1_0_oneprovider_beta_onedata_org/
-
2.0: pre-configured Oneprovider with pre-configured Onezone <br > Sources: scenarios/2_0_oneprovider_onezone/
-
2.1: manual configuration of Oneprovider and Onezone <br > Sources: scenarios/2_1_oneprovider_onezone_onepanel/
-
3.0: Oneprovider with Onezone on separate machines <br > Sources: scenarios/3_0_oneprovider_onezone_multihost/
-
3.1: manual configuration of Oneprovider and Onezone on separate machines <br > Sources: scenarios/3_0_oneprovider_onezone_onepanel_multihost/
If you are new to Onedata please start with scenario 2.0.
-
All scenarios are prepared as Docker Compose configurations.
docker => 1.11 docker-compose => 1.7 -
Depending on the scenario, you might need to create an account on beta.onedata.org.
-
Most scenarios require that your machine has a public IP address assigned to it with a number of open ports open and/or registered domain name to configure working OpenId login and make https work without warnings. Only scenarios 2.0 and 2.1 are designed to work on localhost.
For each scenario, navigate to a scenario directory and execute ./run_onedata.sh script from there.
Onedata services depend on each other. Maintain the order of starting up services and always wait for a message confirming that the service has successfully started.
run_onedata.sh script runs in foreground. To run more complex scenarios, you will need multiple terminal windows or terminal multiplexer such as screen or tmux.
After completing each scenario, you are encouraged to test your installation according to these instructions.
In this scenario you will run a demo of single node pre-configured Oneprovider instance that will register at beta.onedata.orgzone. The configuration template is stored in the docker-compose-oneprovider.yml file and Oneprovider will be automatically installed based on these settings.
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌ ─ ─ ─ ─ ─ ─ ┬ ─ ┬ ─ ─
Onezone: │ Oneprovider │
│beta.onedata.org ◀ ─ ─ ─ ─ ─ ┼▶ │ │
│ ─ ─ ─ ─ ─ ─ ─ │
│ │ Docker Engine│
│ ─ ─ ─ ─ ─ ─ ─ ─ ─ VM1│
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
You will require a machine with a public IP address and a number of open ports to communicate with beta.onedata.org.
In order to setup and deploy your Oneprovider and connect it to beta.onedata.org simply run:
./run_onedata.sh --provider --provider-fqdn <public ip or FQDN of your machine>and wait until it successfully starts.
When Oneprovider is up, you can check it's administration panel at:
https://<public ip or FQDN of your machine>:9443
Now you can test you installation by following these instructions. For further instructions on using Oneprovider refer to documentation on beta.onedata.org.
In this scenario you will run a demo of a fully functional, isolated Onedata deployment that will consist of:
- a single node pre-configured Onezone instance
- a single node pre-configured Oneprovider instance
both running on a single machine.
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┬ ─ ─ ─ ─ ─ ─ ┬ ─ ┬ ─ ┐
│ │ Onezone│ Oneprovider
│ ◀ ─ ─ ─ ─ ─ ┼▶ │ │ │
│ └ ─ ─ ─ ─ ─ ─ ┘ ─ ─ ─ ─ ─ ─ ─
│ Docker Engine │ │
VM1 └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┘
The configuration templates are placed in the docker-compose-oneprovider.yml and docker-compose-onezone.yml files respectively. Oneprovider and Onezone will be automatically installed based on these configuration files.
This scenario assumes that both Docker containers: one with Onezone and the other with Oneprovider will be running on the same machine, thus there is no need for public IP address nor you need to open any ports.
To access those services, you will need to able to reach those containers from Docker's private network. If you run this on your laptop/workstation then its as straightforward as copying a container's address to your web browser. However, if you run them on remote VM you will need to be able to access those ports from your laptop/workstation. We prepared few suggestions how you can reach them.
In order to setup and run your Onedata deployment start Onezone service first:
./run_onedata.sh --zone # In 1st terminal windowwait until it completes successfully, and then start Oneprovider service:
./run_onedata.sh --provider # In 2nd terminal windowThe installation output will provide you with Docker container IP addresses for Onezone and Oneprovider. Detailed information on accessing and using Onedata services can be found here. In order to test your installation please follow these instructions.
In this scenario you will run a demo of a fully functional isolated Onedata installation that will consist of:
- a single node Onezone instance
- a single node Oneprovider instance
both running on a single machine.
This scenario is similar to scenario 2.0. You are adviced to try complete scenario 2.0 before continuing. The only difference is that we did not provide template configuration.
Start Onezone service first:
./run_onedata.sh --zone # In 1st terminal windowwait until it finishes successfully, and then run the Oneprovider service:
./run_onedata.sh --provider # In 2nd terminal windowYou will need to use Onedata web administration tool - Onepanel - to setup Onezone and Oneprovider. You can do that by accessing Onepanel as explained here.
In this scenario you will run a demo of a fully functional isolated onedata deployment that will consist of:
- a single node Onezone instance
- a single node Oneprovider instance
both running on different machines.
┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ┌ ─ ─ ─ ─ ─ ─ ┬ ─ ┬ ─ ─
│ │ Onezone│ Oneprovider │
│ ◀ ─ ─ ─ ─ ─ ┼▶ │ │
│ └ ─ ─ ─ ─ ─ ─ ┤ ─ ─ ─ ─ ─ ─ ─ │
│ Docker Engine │ Docker Engine│
VM1 └ ─ ─ ─ ─ ─ ─ ─ ─ ┤ ─ ─ ─ ─ ─ ─ ─ ─ ─ VM2│
└ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─
The configuration templates are placed in the docker-compose-oneprovider.yml and docker-compose-onezone.yml files respectively. Oneprovider and Onezone will be automatically installed based on these configuration files.
You need to make sure that those ports are accessible between these machines.
To run Onezone and Oneprovider, start Onezone service on the first machine:
./run_onedata.sh --zonewait until it finishes sucessfully and then run Oneprovider on the second machine:
./run_onedata.sh --provider --provider-fqdn <public ip or FQDN of your machine running provider> --zone-fqdn <public ip or FQDN of your machine running zone>Detailed information on accessing and using Onedata services can be found here. In order to test your installation please follow these instrcutions.
In this scenario you will run a demo of a fully functional isolated Onedata deployment that will consist of:
- a single node Onezone instance
- a single node Oneprovider instance
both running on different machines.
You need to make sure that those ports are accessible between those machines.
This scenario is similar to scenario 3.0. You are advised to try complete scenario 3.0 before continuing. The only difference is that we did not provide template configuration.
First start Onezone on the first machine:
./run_onedata.sh --zone when it has started successfully, start Oneprovider on the second machine:
./run_onedata.sh --provider # On second machineYou will need to use Onedata web administration tool - Onepanel - to setup Onezone and Oneprovider. You can do that by accessing Onepanel as explained here.
Detailed information on accessing and using Onedata services can be found here. In order to test your installation please follow these instructions.
If you want (usually you do) your Oneprovider/Onezone to communicate with any Onedata service that is located outside your localhost, you need to open a number of ports:
| Port | Description |
|---|---|
| 53/TCP | DNS (Optional - used for load-balancing) |
| 53/UDP | DNS (Optional - used for load-balancing) |
| 80/TCP | HTTP |
| 443/TCP | HTTPS |
| 5555/TCP | Communication between Oneclient command line tool and Oneprovider service (TCP) |
| 5556/TCP | Communication between Oneprovider services among different sites |
| 6665/TCP | Onedata data transfer channel (RTransfer) |
| 6666/TCP | Onedata data transfer channel (RTransfer) |
| 7443/TCP | Communication between Oneprovider instances and Onezone used to exchange metadata |
| 8443/TCP | REST and CDMI API's (HTTP) |
| 8876/TCP | RTransfer protocol gateway |
| 8877/TCP | RTransfer protocol gateway |
| 9443/TCP | Onepanel web interface |
and make sure that there are no intermediate firewalls blocking those ports between machines running your Onedata services. More information on firewall setup can be found in documentation.
Onezone and Oneprovider offer a web based configuration and management web interface - Onepanel - which can be accessed on port 9443 of the machine running Onedata service. Please note that, if you are running Onezone and Oneprovider on separate machines, Onepanel instances will be deployed on each of these machines.
https://<onezone or oneprovider host IP>:9443
Onepanel plays important function in Oneprovider, as administrators can use it to support users spaces.
The default credentials for accessing Onepanel are:
user: admin
password: password
In addition to web based interface, Onepanel can be also managed using its REST API.
After you have setup Onedata (depending on the scenario) you will have access to:
Oneprovider Onepanel management interface:
https://<ip or domain of your Oneprovider instance>:9443
Onezone Onepanel management interface:
https://<ip or domain if your Onezone instance>:9443
Onezone Portal
https://<ip or domain if your Onezone instance>
Use a basic authentication method to login into Onezone if you haven't configured OpenID. The default credentials to Onezone and to Onepanel are:
user: admin
password: Password1
The basic test of your installatin involves:
- Logging into Onezone
- Getting Space support token for your home space
- Logging into Oneprovider management interface, see accessing onedata services
- Using a token to support your home space, see space support
- Accessing your space in Onezone
- Uploading a file to your space
For more detailed description on how to perform these steps please refer to our official documentation.
TODO. For now please refer to documentation.
Onedata uses OpenID to authenticate with users. However to use that feature you need to:
- register your own domain (if you don't have one, see a short guide how to get free domains)
- register this domain with OpenID providers supported by Onedata, see short guide
From every OpenID provider you will get a pair of (app_id, app_secret). We prepared an example config for you, where you need to replace those values.
After creating your own auth.conf file you need to add a line in volumes section in your docker-compose-onezone.yml. Here is en example:
services:
node1.onezone.localhost:
image: onedata/onezone:some_version
hostname: node1.onezone.localhost
volumes:
- "<absolute path to your auth.config>:/volumes/persistence/var/lib/oz_worker/auth.config" # this line was added
- "${ONEZONE_CONFIG_DIR}:/volumes/persistence"
- "/var/run/docker.sock:/var/run/docker.sock"
After that you can navigate to a domain you registered for you Onezone instance and login with OpenID providers you registered your domain with.
In all scenarios except 2.0 and 2.1, you may want to have your Oneprovider/Onedata installations accessible under public domain to leverage https and OpenID. We recommend that for the purpose of testing you register your own free .tk domain. Such domain can be easily acquired from e.g. http://www.dot.tk/en/index.html. Make this domain point to IP address of your Onezone/Oneprovider machines.
In all scenarios Onezone and Oneprovier services typically have two Docker volumes exported mounted to the host filesystem:
/volumes/persistence- where Onedata configuration is stored/volumes/storage- where Oneprovider stores users' data
Above directories are present in Docker container and are by default mounted under the following paths:
./config_onezonefor Onezone configuration./config_oneproviderfor Oneprovider configuration./oneprovider_datafor Oneprovider users' data
You can modify them directly in Docker Compose configuration files or by using special flags of onedata_run.sh
--provider-data-dirto set directory where provider will store users raw data--provider-conf-dirto set a directory where provider will configuration its files
Additionally we provide a --set-lat-log that tries to deduce the latitude and longitude of you machine. Those coordinates are used in Onezone to display your provider on a world map. This flag works with all scenarios except those that include manual Onepanel configuration process (2.1 and 3.1).
Example execution of Oneprovider with all options:
./onedata_run.sh --provider --provider-fqdn 'myonedataprovider.tk' --zone-fqdn 'myonezone.tk' --provider-data-dir '/mnt/super_fast_big_storage/' --provider-conf-dir '/etc/oneprovider/' --set-lat-logrun_onedata.sh tries to inform you if there are any leftocer configuration or data that might interfere with setting up a fresh Onezone or Oneprovider service. If for some your installation fails, before filing an issue please try to manually remove all contents of those volumes that Oneprovider/Onezone service use.
If you did not use run_onedata.sh with flags such as --provider-conf-dir or --provider-data-dir, hence relying od default directories the git statuscommand should assist you with removing all unnecessary files. If all fails please delete this repo and clone it again.
In each scenario you will deploy a Oneprovider which can be used to support your space. If you are not familiar with the concept of Spaces read the Overview and Space support sections in the documentation. After supporting your space you will be able to access them using a web-interface or Oneclient.
Refer to the documentation of the web interface for further instructions.
In oneclient directory you will find run_oneclient.sh script that will assist you in using Oneclient binary as a docker container.
Example invocation:
./run_oneclient.sh --provider myprovider.tk --token '_Us_MYaSD80YgPpcKfVSLP-Mz3TIqmN1q1vb3qFJ'