Skip to content
This repository has been archived by the owner on Oct 31, 2019. It is now read-only.

Commit

Permalink
More fixes to vignettes #76
Browse files Browse the repository at this point in the history
  • Loading branch information
@andrie
andrie committed Jun 16, 2017
1 parent 38eace0 commit 6d95552
Show file tree
Hide file tree
Showing 4 changed files with 44 additions and 68 deletions.
14 changes: 4 additions & 10 deletions inst/doc/tutorial.R
View file
Original file line number Diff line number Diff line change
Expand Up @@ -48,7 +48,7 @@
# azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1")

## ---- eval=FALSE---------------------------------------------------------
# azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme")
# azureCreateStorageContainer(sc, "opendata", storageAccount = "testmystorage1", resourceGroup = "testme")

## ---- eval=FALSE---------------------------------------------------------
# azureListStorageContainers(sc, storageAccount = "testmystorage1", resourceGroup = "testme")
Expand Down Expand Up @@ -114,12 +114,6 @@
# azureHiveSQL(sc,
# CMD = "select * from hivesampletable",
# path = "wasb://opendata@testmystorage1.blob.core.windows.net/")
#
# azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "opendata")
#
# stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout")
#
# read.delim(text = stdout, header = TRUE, fill = TRUE)

## ---- eval=FALSE---------------------------------------------------------
# azureSparkNewSession(sc, clustername = "smrhdi",
Expand Down Expand Up @@ -153,7 +147,7 @@
# ## [1] "Pi is roughly 3.140285"

## ---- eval=FALSE---------------------------------------------------------
# azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID="0")
# azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID = "0")
#
# #[1] "3.1422"

Expand All @@ -162,6 +156,6 @@
# hdiAdmin = "hdiadmin",
# hdiPassword = "AzureSMR_password123",
# kind = "sparkr")
# azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2")
# azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2")
# azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID = "2")
# azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID = "2")

32 changes: 13 additions & 19 deletions inst/doc/tutorial.Rmd
View file
Original file line number Diff line number Diff line change
Expand Up @@ -114,7 +114,7 @@ azureStopVM(sc, vmName = "DSVM1")

## Accessing storage blobs using the `azureActiveContext`

To access storage blobs you need to have a key. Use `azureSAGetKey()` to automatically retrieve your key or alternatively supply your own key.
To access storage blobs you need to have a key. You can use `azureSAGetKey()` to automatically retrieve your key.


```{r, eval=FALSE}
Expand All @@ -124,7 +124,7 @@ azureSAGetKey(sc, resourceGroup = "testme", storageAccount = "testmystorage1")
To create containers in a storage account use`azureCreateStorageContainer()`

```{r, eval=FALSE}
azureCreateStorageContainer(sc, "opendata",storageAccount = "testmystorage1", resourceGroup = "testme")
azureCreateStorageContainer(sc, "opendata", storageAccount = "testmystorage1", resourceGroup = "testme")
```


Expand Down Expand Up @@ -159,7 +159,7 @@ azureGetBlob(sc, storageAccount = "testmystorage1", container = "opendata",

## Accessing storage blobs without an `azureActiveContext`

It is also possible to access the storage API without having an Azure Active Directory application.
It is also possible to access the blob functionswithout having an Azure Active Directory application.

In this case, you should use the argument `azureActiveContect = NULL` to the storage functions.

Expand All @@ -170,9 +170,9 @@ azureListStorageBlobs(NULL, storageAccount = "testmystorage1", container = "open
```


## Manage HDInsight Clusters
## Manage HDInsight clusters

You can use `AzureSMR` to manage HDInsight clusters. To create a cluster use `azureCreateHDI()`.
You can use `AzureSMR` to manage [HDInsight](https://azure.microsoft.com/en-gb/services/hdinsight/) clusters. To create a cluster use `azureCreateHDI()`.

For advanced configurations use Resource Templates (See below).

Expand Down Expand Up @@ -210,11 +210,11 @@ azureResizeHDI(sc, resourceGroup = "testme", clustername = "smrhdi", role="worke
```


## Resource Templates - Create Azure Resources
## Resource templates - create Azure resources

The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates.
The easiest way to create resources on Azure is to use Azure Resource Manager (ARM) templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into `Settings > Automation scripts`. You can find many example templates at https://github.com/Azure/AzureStack-QuickStart-Templates.

To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example Azure blob).
To create a resource using a template in `AzureSMR` use `azureDeployTemplate()`. The template and paramaters must be available in a public URL (for example in Azure blob store), or you can supply these as JSON strings.

```{r, eval=FALSE}
azureDeployTemplate(sc, resourceGroup = "Analytics", deplName = "Deploy1",
Expand All @@ -233,7 +233,7 @@ ADMIN TIP: If a deployment fails, go to the Azure Portal and look at `Activity l

## Hive Functions

These functions facilitate the use of hive jobs on an HDInsight Cluster
You can use these functions to run and manage hive jobs on an HDInsight Cluster.

```{r, eval=FALSE}
azureHiveStatus(sc, clusterName = "smrhdi",
Expand All @@ -243,20 +243,14 @@ azureHiveStatus(sc, clusterName = "smrhdi",
azureHiveSQL(sc,
CMD = "select * from hivesampletable",
path = "wasb://opendata@testmystorage1.blob.core.windows.net/")
azureListStorageBlobs(sc, storageAccount = "testmystorage1", container = "opendata")
stdout <- azureGetBlob(sc, Container = "test", Blob = "stdout")
read.delim(text = stdout, header = TRUE, fill = TRUE)
```


## Spark functions (experimental)

`AzureSMR` provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session.

To create a new Spark session (via Livy) use `azureSparkNewSession()`
To create a new Spark session (via [Livy](https://github.com/cloudera/hue/tree/master/apps/spark/java#welcome-to-livy-the-rest-spark-server)) use `azureSparkNewSession()`

```{r, eval=FALSE}
azureSparkNewSession(sc, clustername = "smrhdi",
Expand Down Expand Up @@ -299,7 +293,7 @@ azureSparkCMD(sc, CMD = pythonCmd, sessionID = "0")
Check Session variables are retained

```{r, eval=FALSE}
azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID="0")
azureSparkCMD(sc, clustername = "smrhdi", CMD = "print Pi", sessionID = "0")
#[1] "3.1422"
```
Expand All @@ -311,7 +305,7 @@ azureSparkNewSession(sc, clustername = "smrhdi",
hdiAdmin = "hdiadmin",
hdiPassword = "AzureSMR_password123",
kind = "sparkr")
azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID="2")
azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID="2")
azureSparkCMD(sc, clustername = "smrhdi", CMD = "HW<-'hello R'", sessionID = "2")
azureSparkCMD(sc, clustername = "smrhdi", CMD = "cat(HW)", sessionID = "2")
```

34 changes: 14 additions & 20 deletions inst/doc/tutorial.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -153,10 +153,10 @@ <h2>Manage Virtual Machines</h2>
</div>
<div id="accessing-storage-blobs-using-the-azureactivecontext" class="section level2">
<h2>Accessing storage blobs using the <code>azureActiveContext</code></h2>
<p>To access storage blobs you need to have a key. Use <code>azureSAGetKey()</code> to automatically retrieve your key or alternatively supply your own key.</p>
<p>To access storage blobs you need to have a key. You can use <code>azureSAGetKey()</code> to automatically retrieve your key.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureSAGetKey</span>(sc, <span class="dt">resourceGroup =</span> <span class="st">&quot;testme&quot;</span>, <span class="dt">storageAccount =</span> <span class="st">&quot;testmystorage1&quot;</span>)</code></pre></div>
<p>To create containers in a storage account use<code>azureCreateStorageContainer()</code></p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureCreateStorageContainer</span>(sc, <span class="st">&quot;opendata&quot;</span>,<span class="dt">storageAccount =</span> <span class="st">&quot;testmystorage1&quot;</span>, <span class="dt">resourceGroup =</span> <span class="st">&quot;testme&quot;</span>)</code></pre></div>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureCreateStorageContainer</span>(sc, <span class="st">&quot;opendata&quot;</span>, <span class="dt">storageAccount =</span> <span class="st">&quot;testmystorage1&quot;</span>, <span class="dt">resourceGroup =</span> <span class="st">&quot;testme&quot;</span>)</code></pre></div>
<p>To list containers in a storage account use <code>azureListContainers()</code></p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureListStorageContainers</span>(sc, <span class="dt">storageAccount =</span> <span class="st">&quot;testmystorage1&quot;</span>, <span class="dt">resourceGroup =</span> <span class="st">&quot;testme&quot;</span>)</code></pre></div>
<p>To write a blob use <code>azurePutBlob()</code></p>
Expand All @@ -172,14 +172,14 @@ <h2>Accessing storage blobs using the <code>azureActiveContext</code></h2>
</div>
<div id="accessing-storage-blobs-without-an-azureactivecontext" class="section level2">
<h2>Accessing storage blobs without an <code>azureActiveContext</code></h2>
<p>It is also possible to access the storage API without having an Azure Active Directory application.</p>
<p>It is also possible to access the blob functionswithout having an Azure Active Directory application.</p>
<p>In this case, you should use the argument <code>azureActiveContect = NULL</code> to the storage functions.</p>
<p>For example:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureListStorageBlobs</span>(<span class="ot">NULL</span>, <span class="dt">storageAccount =</span> <span class="st">&quot;testmystorage1&quot;</span>, <span class="dt">container =</span> <span class="st">&quot;opendata&quot;</span>)</code></pre></div>
</div>
<div id="manage-hdinsight-clusters" class="section level2">
<h2>Manage HDInsight Clusters</h2>
<p>You can use <code>AzureSMR</code> to manage HDInsight clusters. To create a cluster use <code>azureCreateHDI()</code>.</p>
<h2>Manage HDInsight clusters</h2>
<p>You can use <code>AzureSMR</code> to manage <a href="https://azure.microsoft.com/en-gb/services/hdinsight/">HDInsight</a> clusters. To create a cluster use <code>azureCreateHDI()</code>.</p>
<p>For advanced configurations use Resource Templates (See below).</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureCreateHDI</span>(sc,
<span class="dt">resourceGroup =</span> <span class="st">&quot;testme&quot;</span>,
Expand All @@ -205,9 +205,9 @@ <h2>Manage HDInsight Clusters</h2>
## &quot; headnode ( 2 * Standard_D3_v2 ) workernode ( 5 * Standard_D3_v2 ) zookeepernode ( 3 * Medium ) edgenode0 ( 1 * Standard_D4_v2 )&quot; </code></pre></div>
</div>
<div id="resource-templates---create-azure-resources" class="section level2">
<h2>Resource Templates - Create Azure Resources</h2>
<p>The easiest way to create resources on Azure is to use Azure Templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into <code>Settings &gt; Automation scripts</code>. You can find many example templates at <a href="https://github.com/Azure/AzureStack-QuickStart-Templates" class="uri">https://github.com/Azure/AzureStack-QuickStart-Templates</a>.</p>
<p>To create a resource using a template in <code>AzureSMR</code> use <code>azureDeployTemplate()</code>. The template and paramaters must be available in a public URL (for example Azure blob).</p>
<h2>Resource templates - create Azure resources</h2>
<p>The easiest way to create resources on Azure is to use Azure Resource Manager (ARM) templates. To create Azure resources such as HDInsight clusters there can a large quantity of parameters. Resource templates can be built be creating a resource in the Azure Portal and then going into <code>Settings &gt; Automation scripts</code>. You can find many example templates at <a href="https://github.com/Azure/AzureStack-QuickStart-Templates" class="uri">https://github.com/Azure/AzureStack-QuickStart-Templates</a>.</p>
<p>To create a resource using a template in <code>AzureSMR</code> use <code>azureDeployTemplate()</code>. The template and paramaters must be available in a public URL (for example in Azure blob store), or you can supply these as JSON strings.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureDeployTemplate</span>(sc, <span class="dt">resourceGroup =</span> <span class="st">&quot;Analytics&quot;</span>, <span class="dt">deplName =</span> <span class="st">&quot;Deploy1&quot;</span>,
<span class="dt">templateURL =</span> <span class="st">&quot;{TEMPLATEURL}&quot;</span>, <span class="dt">paramURL =</span> <span class="st">&quot;{PARAMURL}&quot;</span>)

Expand All @@ -221,25 +221,19 @@ <h2>Resource Templates - Create Azure Resources</h2>
</div>
<div id="hive-functions" class="section level2">
<h2>Hive Functions</h2>
<p>These functions facilitate the use of hive jobs on an HDInsight Cluster</p>
<p>You can use these functions to run and manage hive jobs on an HDInsight Cluster.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureHiveStatus</span>(sc, <span class="dt">clusterName =</span> <span class="st">&quot;smrhdi&quot;</span>,
<span class="dt">hdiAdmin =</span> <span class="st">&quot;hdiadmin&quot;</span>,
<span class="dt">hdiPassword =</span> <span class="st">&quot;AzureSMR_password123&quot;</span>)

<span class="kw">azureHiveSQL</span>(sc,
<span class="dt">CMD =</span> <span class="st">&quot;select * from hivesampletable&quot;</span>,
<span class="dt">path =</span> <span class="st">&quot;wasb://opendata@testmystorage1.blob.core.windows.net/&quot;</span>)

<span class="kw">azureListStorageBlobs</span>(sc, <span class="dt">storageAccount =</span> <span class="st">&quot;testmystorage1&quot;</span>, <span class="dt">container =</span> <span class="st">&quot;opendata&quot;</span>)

stdout &lt;-<span class="st"> </span><span class="kw">azureGetBlob</span>(sc, <span class="dt">Container =</span> <span class="st">&quot;test&quot;</span>, <span class="dt">Blob =</span> <span class="st">&quot;stdout&quot;</span>)

<span class="kw">read.delim</span>(<span class="dt">text =</span> stdout, <span class="dt">header =</span> <span class="ot">TRUE</span>, <span class="dt">fill =</span> <span class="ot">TRUE</span>)</code></pre></div>
<span class="dt">path =</span> <span class="st">&quot;wasb://opendata@testmystorage1.blob.core.windows.net/&quot;</span>)</code></pre></div>
</div>
<div id="spark-functions-experimental" class="section level2">
<h2>Spark functions (experimental)</h2>
<p><code>AzureSMR</code> provides some functions that allow HDInsight Spark aessions and jobs to be managed within an R Session.</p>
<p>To create a new Spark session (via Livy) use <code>azureSparkNewSession()</code></p>
<p>To create a new Spark session (via <a href="https://github.com/cloudera/hue/tree/master/apps/spark/java#welcome-to-livy-the-rest-spark-server">Livy</a>) use <code>azureSparkNewSession()</code></p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureSparkNewSession</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>,
<span class="dt">hdiAdmin =</span> <span class="st">&quot;hdiadmin&quot;</span>,
<span class="dt">hdiPassword =</span> <span class="st">&quot;AzureSMR_password123&quot;</span>,
Expand Down Expand Up @@ -268,16 +262,16 @@ <h2>Spark functions (experimental)</h2>

## [1] &quot;Pi is roughly 3.140285&quot;</code></pre></div>
<p>Check Session variables are retained</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureSparkCMD</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>, <span class="dt">CMD =</span> <span class="st">&quot;print Pi&quot;</span>, <span class="dt">sessionID=</span><span class="st">&quot;0&quot;</span>)
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureSparkCMD</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>, <span class="dt">CMD =</span> <span class="st">&quot;print Pi&quot;</span>, <span class="dt">sessionID =</span> <span class="st">&quot;0&quot;</span>)

<span class="co">#[1] &quot;3.1422&quot;</span></code></pre></div>
<p>You can also run SparkR sessions</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">azureSparkNewSession</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>,
<span class="dt">hdiAdmin =</span> <span class="st">&quot;hdiadmin&quot;</span>,
<span class="dt">hdiPassword =</span> <span class="st">&quot;AzureSMR_password123&quot;</span>,
<span class="dt">kind =</span> <span class="st">&quot;sparkr&quot;</span>)
<span class="kw">azureSparkCMD</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>, <span class="dt">CMD =</span> <span class="st">&quot;HW&lt;-'hello R'&quot;</span>, <span class="dt">sessionID=</span><span class="st">&quot;2&quot;</span>)
<span class="kw">azureSparkCMD</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>, <span class="dt">CMD =</span> <span class="st">&quot;cat(HW)&quot;</span>, <span class="dt">sessionID=</span><span class="st">&quot;2&quot;</span>)</code></pre></div>
<span class="kw">azureSparkCMD</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>, <span class="dt">CMD =</span> <span class="st">&quot;HW&lt;-'hello R'&quot;</span>, <span class="dt">sessionID =</span> <span class="st">&quot;2&quot;</span>)
<span class="kw">azureSparkCMD</span>(sc, <span class="dt">clustername =</span> <span class="st">&quot;smrhdi&quot;</span>, <span class="dt">CMD =</span> <span class="st">&quot;cat(HW)&quot;</span>, <span class="dt">sessionID =</span> <span class="st">&quot;2&quot;</span>)</code></pre></div>
</div>


Expand Down
Loading

0 comments on commit 6d95552

Please sign in to comment.