Skip to content

CRAN 0.2.0 checks #19

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Aug 11, 2016
Merged

CRAN 0.2.0 checks #19

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion .Rbuildignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,4 +3,5 @@
^\.httr-oauth$
^\.travis\.yml$
^CONTRIBUTING\.md$
^cran-comments\.md$
^cran-comments\.md$
^revdep
8 changes: 4 additions & 4 deletions DESCRIPTION
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,14 +1,14 @@
Package: bigQueryR
Title: Interface with Google BigQuery with Shiny Compatibility
Version: 0.1.0.9000
Version: 0.2.0
Authors@R: c(person("Mark", "Edmondson",email = "r@sunholo.com",
role = c("aut", "cre")),
person("Hadley", "Wickham", , "hadley@rstudio.com", role = "ctb")
)
Description: Interface with Google BigQuery,
Description: Interface with 'Google BigQuery',
see <https://cloud.google.com/bigquery/> for more information.
This package uses 'googleAuthR' so is compatible with similar packages,
including Google Cloud Storage (<https://cloud.google.com/storage/>) for result extracts.
including 'Google Cloud Storage' (<https://cloud.google.com/storage/>) for result extracts.
URL: http://code.markedmondson.me/bigQueryR/
BugReports: https://github.com/cloudyr/bigQueryR/issues
License: MIT + file LICENSE
Expand All @@ -17,7 +17,7 @@ Depends:
R (>= 3.2.0)
Imports:
googleAuthR (>= 0.3.1),
googleCloudStorageR (>= 0.1.0)
googleCloudStorageR
Suggests:
shiny (>= 0.12.1),
jsonlite (>= 1.0),
Expand Down
8 changes: 2 additions & 6 deletions NEWS.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,8 @@
# bigQueryR 0.1.0.9000
# bigQueryR 0.2.0 (CRAN)

* Download asynch queries straight to disk via googleCloudStorageR


# bigQueryR 0.1.0 (CRAN)
# bigQueryR 0.1.0

* Added a `NEWS.md` file to track changes to the package.
* Initial release



12 changes: 7 additions & 5 deletions cran-comments.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,17 +1,19 @@
## Test environments
* local OS X install, R 3.3.0
* ubuntu 12.04 (on travis-ci), R 3.3.0
* ubuntu 12.04 (on travis-ci), R 3.3.1
* win-builder (devel and release)

## R CMD check results

0 errors | 0 warnings | 2 notes
0 errors | 0 warnings | 1 notes

* This is a new release.
* Possibly mis-spelled words in DESCRIPTION: BigQuery - this is spelt correctly.
* Possibly mis-spelled words in DESCRIPTION:
BigQuery (2:30)

This is spelt correctly

## Reverse dependencies

This is a new release, so there are no reverse dependencies.
googleAnalyticsR is a dependency, that when checked had 0 errors.

---
Binary file added revdep/checks.rds
View file
Open in desktop
Binary file not shown.
196 changes: 196 additions & 0 deletions vignettes/bigQueryR.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,196 @@
<!DOCTYPE html>

<html xmlns="http://www.w3.org/1999/xhtml">

<head>

<meta charset="utf-8">
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="pandoc" />

<meta name="viewport" content="width=device-width, initial-scale=1">

<meta name="author" content="Mark Edmondson" />

<meta name="date" content="2016-08-11" />

<title>bigQueryR</title>



<style type="text/css">code{white-space: pre;}</style>
<style type="text/css">
div.sourceCode { overflow-x: auto; }
table.sourceCode, tr.sourceCode, td.lineNumbers, td.sourceCode {
margin: 0; padding: 0; vertical-align: baseline; border: none; }
table.sourceCode { width: 100%; line-height: 100%; }
td.lineNumbers { text-align: right; padding-right: 4px; padding-left: 4px; color: #aaaaaa; border-right: 1px solid #aaaaaa; }
td.sourceCode { padding-left: 5px; }
code > span.kw { color: #007020; font-weight: bold; } /* Keyword */
code > span.dt { color: #902000; } /* DataType */
code > span.dv { color: #40a070; } /* DecVal */
code > span.bn { color: #40a070; } /* BaseN */
code > span.fl { color: #40a070; } /* Float */
code > span.ch { color: #4070a0; } /* Char */
code > span.st { color: #4070a0; } /* String */
code > span.co { color: #60a0b0; font-style: italic; } /* Comment */
code > span.ot { color: #007020; } /* Other */
code > span.al { color: #ff0000; font-weight: bold; } /* Alert */
code > span.fu { color: #06287e; } /* Function */
code > span.er { color: #ff0000; font-weight: bold; } /* Error */
code > span.wa { color: #60a0b0; font-weight: bold; font-style: italic; } /* Warning */
code > span.cn { color: #880000; } /* Constant */
code > span.sc { color: #4070a0; } /* SpecialChar */
code > span.vs { color: #4070a0; } /* VerbatimString */
code > span.ss { color: #bb6688; } /* SpecialString */
code > span.im { } /* Import */
code > span.va { color: #19177c; } /* Variable */
code > span.cf { color: #007020; font-weight: bold; } /* ControlFlow */
code > span.op { color: #666666; } /* Operator */
code > span.bu { } /* BuiltIn */
code > span.ex { } /* Extension */
code > span.pp { color: #bc7a00; } /* Preprocessor */
code > span.at { color: #7d9029; } /* Attribute */
code > span.do { color: #ba2121; font-style: italic; } /* Documentation */
code > span.an { color: #60a0b0; font-weight: bold; font-style: italic; } /* Annotation */
code > span.cv { color: #60a0b0; font-weight: bold; font-style: italic; } /* CommentVar */
code > span.in { color: #60a0b0; font-weight: bold; font-style: italic; } /* Information */
</style>



<link href="data:text/css;charset=utf-8,body%20%7B%0Abackground%2Dcolor%3A%20%23fff%3B%0Amargin%3A%201em%20auto%3B%0Amax%2Dwidth%3A%20700px%3B%0Aoverflow%3A%20visible%3B%0Apadding%2Dleft%3A%202em%3B%0Apadding%2Dright%3A%202em%3B%0Afont%2Dfamily%3A%20%22Open%20Sans%22%2C%20%22Helvetica%20Neue%22%2C%20Helvetica%2C%20Arial%2C%20sans%2Dserif%3B%0Afont%2Dsize%3A%2014px%3B%0Aline%2Dheight%3A%201%2E35%3B%0A%7D%0A%23header%20%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0A%23TOC%20%7B%0Aclear%3A%20both%3B%0Amargin%3A%200%200%2010px%2010px%3B%0Apadding%3A%204px%3B%0Awidth%3A%20400px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Aborder%2Dradius%3A%205px%3B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Afont%2Dsize%3A%2013px%3B%0Aline%2Dheight%3A%201%2E3%3B%0A%7D%0A%23TOC%20%2Etoctitle%20%7B%0Afont%2Dweight%3A%20bold%3B%0Afont%2Dsize%3A%2015px%3B%0Amargin%2Dleft%3A%205px%3B%0A%7D%0A%23TOC%20ul%20%7B%0Apadding%2Dleft%3A%2040px%3B%0Amargin%2Dleft%3A%20%2D1%2E5em%3B%0Amargin%2Dtop%3A%205px%3B%0Amargin%2Dbottom%3A%205px%3B%0A%7D%0A%23TOC%20ul%20ul%20%7B%0Amargin%2Dleft%3A%20%2D2em%3B%0A%7D%0A%23TOC%20li%20%7B%0Aline%2Dheight%3A%2016px%3B%0A%7D%0Atable%20%7B%0Amargin%3A%201em%20auto%3B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dcolor%3A%20%23DDDDDD%3B%0Aborder%2Dstyle%3A%20outset%3B%0Aborder%2Dcollapse%3A%20collapse%3B%0A%7D%0Atable%20th%20%7B%0Aborder%2Dwidth%3A%202px%3B%0Apadding%3A%205px%3B%0Aborder%2Dstyle%3A%20inset%3B%0A%7D%0Atable%20td%20%7B%0Aborder%2Dwidth%3A%201px%3B%0Aborder%2Dstyle%3A%20inset%3B%0Aline%2Dheight%3A%2018px%3B%0Apadding%3A%205px%205px%3B%0A%7D%0Atable%2C%20table%20th%2C%20table%20td%20%7B%0Aborder%2Dleft%2Dstyle%3A%20none%3B%0Aborder%2Dright%2Dstyle%3A%20none%3B%0A%7D%0Atable%20thead%2C%20table%20tr%2Eeven%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Ap%20%7B%0Amargin%3A%200%2E5em%200%3B%0A%7D%0Ablockquote%20%7B%0Abackground%2Dcolor%3A%20%23f6f6f6%3B%0Apadding%3A%200%2E25em%200%2E75em%3B%0A%7D%0Ahr%20%7B%0Aborder%2Dstyle%3A%20solid%3B%0Aborder%3A%20none%3B%0Aborder%2Dtop%3A%201px%20solid%20%23777%3B%0Amargin%3A%2028px%200%3B%0A%7D%0Adl%20%7B%0Amargin%2Dleft%3A%200%3B%0A%7D%0Adl%20dd%20%7B%0Amargin%2Dbottom%3A%2013px%3B%0Amargin%2Dleft%3A%2013px%3B%0A%7D%0Adl%20dt%20%7B%0Afont%2Dweight%3A%20bold%3B%0A%7D%0Aul%20%7B%0Amargin%2Dtop%3A%200%3B%0A%7D%0Aul%20li%20%7B%0Alist%2Dstyle%3A%20circle%20outside%3B%0A%7D%0Aul%20ul%20%7B%0Amargin%2Dbottom%3A%200%3B%0A%7D%0Apre%2C%20code%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0Aborder%2Dradius%3A%203px%3B%0Acolor%3A%20%23333%3B%0Awhite%2Dspace%3A%20pre%2Dwrap%3B%20%0A%7D%0Apre%20%7B%0Aborder%2Dradius%3A%203px%3B%0Amargin%3A%205px%200px%2010px%200px%3B%0Apadding%3A%2010px%3B%0A%7D%0Apre%3Anot%28%5Bclass%5D%29%20%7B%0Abackground%2Dcolor%3A%20%23f7f7f7%3B%0A%7D%0Acode%20%7B%0Afont%2Dfamily%3A%20Consolas%2C%20Monaco%2C%20%27Courier%20New%27%2C%20monospace%3B%0Afont%2Dsize%3A%2085%25%3B%0A%7D%0Ap%20%3E%20code%2C%20li%20%3E%20code%20%7B%0Apadding%3A%202px%200px%3B%0A%7D%0Adiv%2Efigure%20%7B%0Atext%2Dalign%3A%20center%3B%0A%7D%0Aimg%20%7B%0Abackground%2Dcolor%3A%20%23FFFFFF%3B%0Apadding%3A%202px%3B%0Aborder%3A%201px%20solid%20%23DDDDDD%3B%0Aborder%2Dradius%3A%203px%3B%0Aborder%3A%201px%20solid%20%23CCCCCC%3B%0Amargin%3A%200%205px%3B%0A%7D%0Ah1%20%7B%0Amargin%2Dtop%3A%200%3B%0Afont%2Dsize%3A%2035px%3B%0Aline%2Dheight%3A%2040px%3B%0A%7D%0Ah2%20%7B%0Aborder%2Dbottom%3A%204px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Apadding%2Dbottom%3A%202px%3B%0Afont%2Dsize%3A%20145%25%3B%0A%7D%0Ah3%20%7B%0Aborder%2Dbottom%3A%202px%20solid%20%23f7f7f7%3B%0Apadding%2Dtop%3A%2010px%3B%0Afont%2Dsize%3A%20120%25%3B%0A%7D%0Ah4%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23f7f7f7%3B%0Amargin%2Dleft%3A%208px%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Ah5%2C%20h6%20%7B%0Aborder%2Dbottom%3A%201px%20solid%20%23ccc%3B%0Afont%2Dsize%3A%20105%25%3B%0A%7D%0Aa%20%7B%0Acolor%3A%20%230033dd%3B%0Atext%2Ddecoration%3A%20none%3B%0A%7D%0Aa%3Ahover%20%7B%0Acolor%3A%20%236666ff%3B%20%7D%0Aa%3Avisited%20%7B%0Acolor%3A%20%23800080%3B%20%7D%0Aa%3Avisited%3Ahover%20%7B%0Acolor%3A%20%23BB00BB%3B%20%7D%0Aa%5Bhref%5E%3D%22http%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0Aa%5Bhref%5E%3D%22https%3A%22%5D%20%7B%0Atext%2Ddecoration%3A%20underline%3B%20%7D%0A%0Acode%20%3E%20span%2Ekw%20%7B%20color%3A%20%23555%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Edt%20%7B%20color%3A%20%23902000%3B%20%7D%20%0Acode%20%3E%20span%2Edv%20%7B%20color%3A%20%2340a070%3B%20%7D%20%0Acode%20%3E%20span%2Ebn%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Efl%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Ech%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Est%20%7B%20color%3A%20%23d14%3B%20%7D%20%0Acode%20%3E%20span%2Eco%20%7B%20color%3A%20%23888888%3B%20font%2Dstyle%3A%20italic%3B%20%7D%20%0Acode%20%3E%20span%2Eot%20%7B%20color%3A%20%23007020%3B%20%7D%20%0Acode%20%3E%20span%2Eal%20%7B%20color%3A%20%23ff0000%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%0Acode%20%3E%20span%2Efu%20%7B%20color%3A%20%23900%3B%20font%2Dweight%3A%20bold%3B%20%7D%20%20code%20%3E%20span%2Eer%20%7B%20color%3A%20%23a61717%3B%20background%2Dcolor%3A%20%23e3d2d2%3B%20%7D%20%0A" rel="stylesheet" type="text/css" />

</head>

<body>




<h1 class="title toc-ignore">bigQueryR</h1>
<h4 class="author"><em>Mark Edmondson</em></h4>
<h4 class="date"><em>2016-08-11</em></h4>



<div id="introduction" class="section level2">
<h2>Introduction</h2>
<p>This is a package for interating with <a href="https://cloud.google.com/bigquery/">BigQuery</a> from within R.</p>
<p>You may want instead to use <a href="https://github.com/hadley/bigrquery">bigrquery</a> which is more developed with integration with <code>dplyr</code> etc. Some functions from <code>bigrquery</code> are used in this package.</p>
<div id="why-this-package-then" class="section level3">
<h3>Why this package then?</h3>
<p>This package is here as it uses <a href="https://github.com/MarkEdmondson1234/googleAuthR">googleAuthR</a> as backend, so has Shiny support, and compatibility with other googleAuthR dependent packages.</p>
<p>It also has support for data extracts to Google Cloud Storage, meaning you can download data and make the download URL available to a user via their Google email. If you do a query normally with over 100000 results it hangs and errors.</p>
<p>An example of a BigQuery Shiny app running OAuth2 is here, the <a href="https://mark.shinyapps.io/bigquery-viz/">BigQuery Visualiser</a></p>
</div>
</div>
<div id="authentication" class="section level2">
<h2>Authentication</h2>
<p>Authentication is as used in other <code>googleAuthR</code> libraries:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(bigQueryR)

## this will open your browser
## Authenticate with an email that has access to the BigQuery project you need
<span class="kw">bqr_auth</span>()

## verify under a new user
<span class="kw">bqr_auth</span>(<span class="dt">new_user=</span><span class="ot">TRUE</span>)</code></pre></div>
<p>If you are authenticating under several APIs via <code>googleAuthR</code>then use <code>gar_auth()</code> instead with the appropriate scopes set.</p>
<p>You can also use service-to-service JSON files and multi-user authentication under Shiny, see the <code>googleAuthR</code> readme for details.</p>
</div>
<div id="listing-bigquery-meta-data" class="section level2">
<h2>Listing BigQuery meta data</h2>
<p>Various functions for listing what is in your BigQuery account.</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(bigQueryR)

## this will open your browser
## Authenticate with an email that has access to the BigQuery project you need
<span class="kw">bqr_auth</span>()

## verify under a new user
<span class="kw">bqr_auth</span>(<span class="dt">new_user=</span><span class="ot">TRUE</span>)

## get projects
projects &lt;-<span class="st"> </span><span class="kw">bqr_list_projects</span>()

my_project &lt;-<span class="st"> </span>projects[<span class="dv">1</span>]

## for first project, get datasets
datasets &lt;-<span class="st"> </span>bqr_list_datasets[my_project]

my_dataset &lt;-<span class="st"> </span>datasets[<span class="dv">1</span>]
## list tables
my_table &lt;-<span class="st"> </span><span class="kw">bqr_list_tables</span>(my_project, my_dataset)

## get metadata for table
meta_table &lt;-<span class="st"> </span><span class="kw">bqr_table_meta</span>(my_project, my_dataset, my_table)</code></pre></div>
</div>
<div id="simple-queries" class="section level2">
<h2>Simple Queries</h2>
<p>You can pass in queries that have results under ~ 100000 rows using this command:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">bqr_query</span>(<span class="st">&quot;big-query-r&quot;</span>,<span class="st">&quot;samples&quot;</span>,
<span class="st">&quot;SELECT COUNT(repository.url) FROM [publicdata:samples.github_nested]&quot;</span>)</code></pre></div>
<p>More than that, and the API starts to hang and you are limited by your download bandwidth.</p>
</div>
<div id="asynchronous-queries" class="section level2">
<h2>Asynchronous Queries</h2>
<p>For bigger queries, asynchronous queries save the results to another BigQuery table. You can check the progress of the job via <code>bqr_get_job</code></p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r"><span class="kw">library</span>(bigQueryR)

## Auth with a project that has at least BigQuery and Google Cloud Storage scope
<span class="kw">bqr_auth</span>()

## make a big query
job &lt;-<span class="st"> </span><span class="kw">bqr_query_asynch</span>(<span class="st">&quot;your_project&quot;</span>,
<span class="st">&quot;your_dataset&quot;</span>,
<span class="st">&quot;SELECT * FROM blah LIMIT 9999999&quot;</span>,
<span class="dt">destinationTableId =</span> <span class="st">&quot;bigResultTable&quot;</span>)

## poll the job to check its status
## its done when job$status$state == &quot;DONE&quot;
<span class="kw">bqr_get_job</span>(<span class="st">&quot;your_project&quot;</span>, job$jobReference$jobId)

##once done, the query results are in &quot;bigResultTable&quot;</code></pre></div>
<p>You may now want to download this data. For large datasets, this is best done via extracting the BigQuery result to Google Cloud Storage, then downloading the data from there.</p>
<p>You can create a bucket at Google Cloud Storage see <a href="https://cloud.google.com/storage/docs/cloud-console" class="uri">https://cloud.google.com/storage/docs/cloud-console</a>, or you can use <a href="https://github.com/cloudyr/googleCloudStorageR">library(googleCloudStorageR)</a></p>
<p>Once created, you can extract your data via the below:</p>
<div class="sourceCode"><pre class="sourceCode r"><code class="sourceCode r">## Create the data extract from BigQuery to Cloud Storage
job_extract &lt;-<span class="st"> </span><span class="kw">bqr_extract_data</span>(<span class="st">&quot;your_project&quot;</span>,
<span class="st">&quot;your_dataset&quot;</span>,
<span class="st">&quot;bigResultTable&quot;</span>,
<span class="st">&quot;your_cloud_storage_bucket_name&quot;</span>)

## poll the extract job to check its status
## its done when job$status$state == &quot;DONE&quot;
<span class="kw">bqr_get_job</span>(<span class="st">&quot;your_project&quot;</span>, job_extract$jobReference$jobId)

## to download via a URL and not logging in via Google Cloud Storage interface:
## Use an email that is Google account enabled
## Requires scopes:
## https://www.googleapis.com/auth/devstorage.full_control
## https://www.googleapis.com/auth/cloud-platform
## set via options(&quot;bigQueryR.scopes&quot;) and reauthenticate if needed

download_url &lt;-<span class="st"> </span><span class="kw">bqr_grant_extract_access</span>(job_extract, <span class="st">&quot;your@email.com&quot;</span>)

## download_url may be multiple if the data is &gt; 1GB
&gt;<span class="st"> </span>[<span class="dv">1</span>] <span class="st">&quot;https://storage.cloud.google.com/big-query-r-extracts/extract-20160311112410-000000000000.csv&quot;</span>
&gt;<span class="st"> </span>[<span class="dv">2</span>] <span class="st">&quot;https://storage.cloud.google.com/big-query-r-extracts/extract-20160311112410-000000000001.csv&quot;</span>
&gt;<span class="st"> </span>[<span class="dv">3</span>] <span class="st">&quot;https://storage.cloud.google.com/big-query-r-extracts/extract-20160311112410-000000000002.csv&quot;</span></code></pre></div>
</div>



<!-- dynamically load mathjax for compatibility with self-contained -->
<script>
(function () {
var script = document.createElement("script");
script.type = "text/javascript";
script.src = "https://cdn.mathjax.org/mathjax/latest/MathJax.js?config=TeX-AMS-MML_HTMLorMML";
document.getElementsByTagName("head")[0].appendChild(script);
})();
</script>

</body>
</html>