Skip to content
This repository has been archived by the owner on Feb 3, 2021. It is now read-only.

Commit

Permalink
Feature: first run docs update (#644)
Browse files Browse the repository at this point in the history
* update getting started page, remove custom scripts doc

* recommend venv
  • Loading branch information
jafreck authored Aug 17, 2018
1 parent b7bdd8c commit 9098533
Show file tree
Hide file tree
Showing 4 changed files with 105 additions and 157 deletions.
153 changes: 105 additions & 48 deletions docs/00-getting-started.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,45 +2,110 @@
The minimum requirements to get started with this package are:
- Python 3.5+, pip 9.0.1+
- An Azure account
- An Azure Batch account
- An Azure Storage account

## Cloning and installing the project
1. Clone the repo
2. Make sure you are running python 3.5 or greater.
_If the default version on your machine is python 2 make sure to run the following commands with **pip3** instead of **pip**._
## Installation
Before you start, ensure you are running python 3.5 or greater by running: `python --version`.

### Install from pip
It is recommended that you install `aztk` in a virtual environment:
```
# install venv
pip install python-venv
# create a virutal environment called env
python -m venv env
# activate the virtual environment (linux)
source env/bin/activate
3. Install `aztk`:
# activate the virtual environment (windows)
env/Scripts/activate
```
To install `aztk` using `pip`, run:
```sh
pip install aztk
```

### Install from source
1. Clone the repo
```sh
git clone https://github.com/Azure/aztk.git
```
2. Install `aztk`:
```sh
pip install -e .
```
5. Initialize your environment:
### Initialize your environment
Navigate to the directory you wish to use as your spark development environment, and run:
```sh
aztk spark init
```
This will create a *.aztk* folder with preset configuration files in your current working directory.

If you would like to initialize your `aztk` clusters with a specific development toolset, please pass one of the following flags:
```bash
aztk spark init --python
aztk spark init --R
aztk spark init --scala
aztk spark init --java
```

Navigate to the directory you wish to use as your spark development environment, and run:
```bash
aztk spark init
```
This will create a *.aztk* folder with preset configuration files in your current working directory.

If you would like to initialize your AZTK clusters with a specific development toolset, please pass one of the following flags:
```bash
aztk spark init --python
aztk spark init --R
aztk spark init --scala
aztk spark init --java
```
If you wish to have global configuration files that will be read regardless of your current working directory, run:
```bash
aztk spark init --global
```
This will put default configuration files in your home directory, *~/*. Please note that configuration files in your current working directory will take precedence over global configuration files in your home directory.

If you wish to have global configuration files that will be read regardless of your current working directory, run:
```bash
aztk spark init --global
```
This will put default configuration files in your home directory, *~/*. Please note that configuration files in your current working directory will take precedence over global configuration files in your home directory.
## Account Setup

To create the necessary Azure Resources, either:
1. [Run the provided account setup script.](#account-setup-script)
2. [Create the resources manually.](#manual-resource-creation)

### Account Setup Script
#### Overview
The account setup script creates and configures all of the required Azure resources.

The script will create and configure the following resources:
- Resource group
- Storage account
- Batch account
- Azure Active Directory application and service principal
<!-- - Virtual network with a configured subnet -->

The script outputs all of the necessary information to use `aztk`, copy the output into the `.aztk/secrets.yaml` file created when running `aztk spark init`.

#### Usage
Copy and paste the following into an [Azure Cloud Shell](https://shell.azure.com):
```sh
wget -q https://raw.githubusercontent.com/Azure/aztk/v0.8.1/account_setup.sh &&
chmod 755 account_setup.sh &&
/bin/bash account_setup.sh
```
A series of prompts will appear, and you can set the values you desire for each field. Default values appear in brackets `[]` and will be used if no value is provided.
```
Azure Region [westus]:
Resource Group Name [aztk]:
Storage Account Name [aztkstorage]:
Batch Account Name [aztkbatch]:
Active Directory Application Name [aztkapplication]:
Active Directory Application Credential Name [aztk]:
```

## Setting up your accounts
Once the script has finished running you will see the following output:

### Using the account setup script
A script to create and configure the Azure resources required to use `aztk` is provided. For more more information and usage, see [Getting Started Script](01-getting-started-script.html)
```
service_principal:
tenant_id: <AAD Diretory ID>
client_id: <AAD App Application ID>
credential: <AAD App Password>
batch_account_resource_id: </batch/account/resource/id>
storage_account_resource_id: </storage/account/resource/id>
```

Copy the entire `service_principal` section in your `.aztk/secrets.yaml`. If you do not have a `secrets.yaml` file, you can create one in your current working directory by running `aztk spark init`.

Now you are ready to create your first `aztk` cluster. See [Creating a Cluster](./10-clusters.html#creating-a-cluster).

### Manual resource creation
To finish setting up, you need to fill out your Azure Batch and Azure Storage secrets in *.aztk/secrets.yaml*. We'd also recommend that you enter SSH key info in this file too.
Expand All @@ -54,14 +119,13 @@ ssh_pub_key: ~/.ssh/my-public-key.pub
ssh_priv_key: ~/.ssh/my-private-key
```
0. Log into Azure
If you do not already have an Azure account, go to [https://azure.microsoft.com/](https://azure.microsoft.com/) to get started for free today.
#### Log into Azure
If you do not already have an Azure account, go to [https://azure.microsoft.com](https://azure.microsoft.com) and create an account.
Once you have one, simply log in and go to the [Azure Portal](https://portal.azure.com) to start creating your Azure Batch account and Azure Storage account.
Once you have one, log in and go to the [Azure Portal](https://portal.azure.com) to create your Azure Batch account and Azure Storage account.
#### Using AAD
To get the required keys for your Azure Active Directory (AAD) Service Principal, Azure Batch Account and Azure Storage Account, please follow these instructions. Note that this is the recommended path for use with AZTK, as some features require AAD and are disabled if using Shared Key authentication.
#### Using Azure Active Directory Authentication
To get the required keys for your Azure Active Directory (AAD) Service Principal, Azure Batch Account and Azure Storage Account, please follow these instructions. Note that this is the recommended path for use with `aztk`, as some features require AAD and are disabled if using the alternative Shared Key authentication.
1. Register an Azure Active Directory (AAD) Application
Expand Down Expand Up @@ -131,12 +195,12 @@ service_principal:
storage_account_resource_id: </storage/account/resource/id>
```

### Using Shared Keys
_Please note that using Shared Keys prevents the use of certain AZTK features including Mixed Mode clusters and support for VNETs._
#### Using Shared Key Authentication
Please note that using Shared Keys prevents the use of certain `aztk` features including low priority nodes and VNET support. It is recommended to use [Azure Active Directory (AAD) Authentication](#using-azure-active-directory-authentication).

To get the required keys for Azure Batch and Azure Storage, please follow the below instructions:

1. Create a Storage account
##### Create a Storage account

- Click the '+' button at the top left of the screen and search for 'Storage'. Select 'Storage account - blob, file, table, queue' and click 'Create'

Expand All @@ -146,7 +210,7 @@ To get the required keys for Azure Batch and Azure Storage, please follow the be

![](./misc/Storage_2.png)

2. Create a Batch account
##### Create a Batch account

- Click the '+' button at the top left of the screen and search for 'Compute'. Select 'Batch' and click 'Create'

Expand All @@ -156,21 +220,14 @@ To get the required keys for Azure Batch and Azure Storage, please follow the be

![](./misc/Batch_2.png)

4. Save your account credentials into the secrets.yaml file
##### Save your account credentials into the secrets.yaml file

- Open the secrets.yaml file in the *.aztk* folder in your current working directory (if *.aztk* doesn't exist, run `aztk spark init`). Fill in all of the fields as described below.
- Go to the accounts in the Azure portal and copy paste the account names, keys and other information needed into the
secrets file.
### Storage account
Open the `.aztk/secrets.yaml` file in your current working directory (if `.aztk/` doesn't exist, [initialize your environment](#initialize-your-environment). Fill in all of the fields as described below.
For the Storage account, copy the name and one of the two keys:
![](./misc/Storage_secrets.png)
### Batch account
For the Batch account, copy the name, the url and one of the two keys:
![](./misc/Batch_secrets.png)
Expand Down
44 changes: 0 additions & 44 deletions docs/01-getting-started-script.md

This file was deleted.

63 changes: 0 additions & 63 deletions docs/11-custom-scripts.md

This file was deleted.

2 changes: 0 additions & 2 deletions docs/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,7 @@ This toolkit is built on top of Azure Batch but does not require any Azure Batch
:caption: User documentation:

00-getting-started
01-getting-started-script
10-clusters
11-custom-scripts
12-docker-image
13-configuration
14-azure-files
Expand Down

0 comments on commit 9098533

Please sign in to comment.