Skip to content

Commit

Permalink
Athena Database Connector (#352)
Browse files Browse the repository at this point in the history
* Added new Datasource install file

* Check in latest instructions

* Check in new file based on comments

* Add icons for Athena

* Updated Instructions

* Check in constants for Athena

* Got UI to detect Athena

* Check in latest version

* Check in first version of schemas

* Check in schema definition

* Check in more documentation for this

* Check in updated new connection

* Check in draft of new connection:

* Check in fixes based on Nicolas comment

* Updated Datastores.js instructions

* Check in small fix

* Check in settings to get SQL

* Check in small update for Settings.js

* Check in first draft of unit tests

* Updating unit tests

* Check in unit tests for connection

* Check in latest tets for database

* Update to do for test cases

* Check in git ignore update

* Check in first function for testing

* Check in basic test to start query

* Check in version for getting connection status

* First stab at execution method

* Fixing Query Results

* Check in latest version

* Check in test information

* Check in latest parsing

* Check in latest connection properties

* Updated the test cases

* Check in latest vesrion with getting table names

* Checkn in cleanup for get tables

* Check in get list of Schemas

* Code Clean up

* Check in fix for unit tests

* Check in latest version

* Check in latest

* Check in latest changes based on @n-riesco comments

* Check in updated new Connection

* Checking in updates for environment and troubleshooting

* Check in fixes based on @n-riesco

* Check in fixes based on @n-riesco code review

* Getting tests back up and running

* Get integration working

* Check in fix for todo

* Put in solution for describing dabases

* Getting column names

* Correctly formating the extracting of rows

* Check in latest value

* Adding in getting by schame names

* Merging db name and table name

* Check in latest

* Renamed database property

* Check in Athena query

* Check in todo

* Check in basic fix for query

* Fix eslint for athena constants and spec

* Automatic Fixing of ESLint

* Check in clean eslint

* Check in notes for TODO

* Check in TODO code

* ESlint fixes after merge from master

* Adding small code cleanup

* Check in small fixes

* Removed todo file as not needed

* First comments on PR

* Check to add fixes

* Fixing up athena driver based on @n-riesco

* Fix lint and unit test

* Check in latest version

* Check in sample SQL and instructions

* Check in existing plotly sample csv

* Check in latest vesrion

* Check in lint and document updates

* Renaming file names with git

* Fixing upper case name

* Code reviews for unit test @n-riesco

* Updates for Plot.ly Athena Config

* Code Cleanup

* Cleanup athena driver for nock unit tests

* Fixes for UI Constants

* Debugging connect

* athena: fix tests

* All tests share the Athena connection.

* Added connect() test and ensured it runs first, so that
  connection.athenaClient is defined for the remaining tests.

* Defined helper function mockAthenaResponses(query, columnNames, rows).

* Fixed tables() test (tables() shouldn't return the column name
  `table_names`).

* athena: fix sample credential

* athena: fix tables and remove unnecessary promises

* Fixed tables() so that it doesn't return the column name table_names.

* Avoided the creation of unnecessary promises.

* Used slice() and map() to generate the response rows.

* Code clean up for Athena Unit Tests

* Check in small fix for @n-riesco
  • Loading branch information
shannonlal committed Apr 10, 2018
1 parent 3ebfd4d commit a2fe8bd
Show file tree
Hide file tree
Showing 12 changed files with 966 additions and 1 deletion.
3 changes: 3 additions & 0 deletions .gitignore
Expand Up @@ -64,3 +64,6 @@ backend/headlessBundle.js
backend/headlessBundle.js.map

npm-shrinkwrap.unsafe.json

#AWS Credentials
aws.json
2 changes: 2 additions & 0 deletions app/components/Settings/Tabs/Tab.react.js
Expand Up @@ -36,6 +36,8 @@ export default class ConnectionTab extends Component {
label = connectionObject.label || connectionObject.id || connectionObject.database;
} else if (connectionObject.dialect === DIALECTS.ELASTICSEARCH) {
label = `Elasticsearch (${connectionObject.host})`;
} else if (connectionObject.dialect === DIALECTS.ATHENA) {
label = `Athena (${connectionObject.database})`;
} else if (connectionObject.dialect === DIALECTS.SQLITE) {
label = connectionObject.storage;
} else if (connectionObject.dialect === DIALECTS.DATA_WORLD) {
Expand Down
45 changes: 44 additions & 1 deletion app/constants/constants.js
Expand Up @@ -16,6 +16,7 @@ export const DIALECTS = {
APACHE_IMPALA: 'apache impala',
APACHE_DRILL: 'apache drill',
DATA_WORLD: 'data.world',
ATHENA: 'athena',
CSV: 'csv'
};

Expand All @@ -30,6 +31,7 @@ export const SQL_DIALECTS_USING_EDITOR = [
'apache spark',
'apache impala',
'data.world',
'athena',
'csv'
];

Expand Down Expand Up @@ -189,6 +191,7 @@ export const CONNECTION_CONFIG = {
'type': 'password'
}
], // TODO - password options for apache drill

[DIALECTS.DATA_WORLD]: [
{
'label': 'Dataset/Project URL',
Expand All @@ -202,6 +205,36 @@ export const CONNECTION_CONFIG = {
'type': 'password',
'description': 'Your data.world read/write token. It can be obtained from https://data.world/settings/advanced'
}
],
[DIALECTS.ATHENA]: [
{
'label': 'S3 Access Key', 'value': 'accessKey', 'type': 'password'
},
{
'label': 'S3 Secret Access Key', 'value': 'secretAccessKey', 'type': 'password'
},
{
'label': 'AWS Region', 'value': 'region', 'type': 'text',
'description': 'The AWS region (i.e. us-east-1) where the database resides'
},
{
'label': 'S3 Bucket', 'value': 'outputS3Bucket', 'type': 'text',
'description': 'The Athena connector will store query results in this location.'
},
{
'label': 'Database', 'value': 'database', 'type': 'text'
},
{
'label': 'Query Interval', 'value': 'queryInterval', 'type': 'number',
'description': 'The Interval (In Milliseconds) that Falcon will check to see \
if the Athena Query is done. Default 2 seconds'
},
{
'label': 'SSL Enabled', 'value': 'sslEnabled', 'type': 'checkbox',
'description': 'Does your database require that you connect to it via SSL? \
Note that this is just the connection between this app and your database; \
connections to plot.ly or your plotly instance are always encrypted.'
}
]
};

Expand All @@ -220,7 +253,8 @@ export const LOGOS = {
[DIALECTS.SQLITE]: 'images/sqlite-logo.png',
[DIALECTS.S3]: 'images/s3-logo.png',
[DIALECTS.APACHE_DRILL]: 'images/apache_drill-logo.png',
[DIALECTS.DATA_WORLD]: 'images/dataworld-logo.png'
[DIALECTS.DATA_WORLD]: 'images/dataworld-logo.png',
[DIALECTS.ATHENA]: 'images/athena-logo.png'
};

export function PREVIEW_QUERY(connection, table, elasticsearchIndex) {
Expand All @@ -237,6 +271,7 @@ export function PREVIEW_QUERY(connection, table, elasticsearchIndex) {
case DIALECTS.POSTGRES:
case DIALECTS.DATA_WORLD:
case DIALECTS.REDSHIFT:
case DIALECTS.ATHENA:
return `SELECT * FROM ${table} LIMIT 1000`;
case DIALECTS.MSSQL:
return `SELECT TOP 1000 * FROM ${connection.database}.dbo.${table}`;
Expand Down Expand Up @@ -411,6 +446,14 @@ export const SAMPLE_DBS = {
dialect: 'sqlite',
storage: `${__dirname}/plotly_datasets.db`
},
[DIALECTS.ATHENA]: {
s3Outputlocation: 'plotly-s3-connector-test',
accessKey: 'AKIAIMHMSHTGARJYSKMQ',
secretAccessKey: 'Urvus4R7MnJOAqT4U3eovlCBimQ4Zg2Y9sV5LWow',
region: 'us-east-1',
database: 'falcon',
queryTimeout: 5000
},
[DIALECTS.DATA_WORLD]: {
url: 'https://data.world/rflprr/reported-lyme-disease-cases-by-state'
}
Expand Down
97 changes: 97 additions & 0 deletions app/images/athena-logo.LICENSE
@@ -0,0 +1,97 @@
Revised September 22, 2017

1. Introduction. These Amazon Web Services (“AWS”) Trademark Guidelines (the “Trademark Guidelines”) form an integral part of the AWS Customer Agreement (the “Agreement”) between Amazon Web Services, Inc. or its affiliates (“AWS,” “we,” “us” or “our”) and you or the entity you represent (“you”). These Trademark Guidelines provide you a limited permission to use the AWS Marks (as defined in Section 2 below), in connection with (as applicable): (a) your use of the Services (as defined in the Agreement), or in connection with software products designed to be used with the Services, or (b) as otherwise agreed with AWS in writing, on the terms set forth herein and in the Agreement, until such time as we may terminate such permission, which we may do at any time, in our sole discretion, as set forth in Section 4 below. The AWS Marks are some of our most valuable assets and these Trademark Guidelines are intended to preserve the value attached to the AWS Marks.

2. Definition. For the purposes of these Trademark Guidelines, "AWS Marks" means the following trademarks, service marks, service or trade names, logos, product names, or designations of AWS and its affiliates: (i) the "Powered by AWS" logo in the two forms shown below (the "Logo"); and (ii) "Amazon Web Services", "AWS", "Amazon[(Chinese characters)] AWS", “Alexa Site Thumbnail”, “Alexa Top Sites”, “Alexa Web Information Service”, “Alexa Web Search”, “Amazon Athena”, “Amazon Aurora”, “Amazon Chime”, “Amazon CloudFront”, “Amazon CloudSearch”, “Amazon CloudWatch”, “Amazon Cognito”, “Amazon Connect”, “Amazon DevPay”, “Amazon DynamoDB”, “Amazon DynamoDB “Accelerator”, “Amazon EC2”, “Amazon Elastic Beanstalk”, “Amazon Elastic Compute Cloud”, “Amazon ElastiCache”, “Amazon Flexible Payments Service”, “Amazon FPS”, “Amazon Fulfillment Web Service”, “Amazon FWS”, “Amazon GameLift”, “Amazon Glacier”, “Amazon Inspector”, “Amazon Kinesis”, “Amazon Lex”, “Amazon Lightsail”, “Amazon Lumberyard”, “Amazon Machine Learning”, “Amazon Macie”, “Amazon Mechanical Turk”, “Amazon Pinpoint”, “Amazon Polly”, “Amazon Quicksight”, “Amazon RDS”, “Amazon Redshift”, “Amazon Rekognition”, “Amazon Relational Database”, “Amazon Route 53”, “Amazon S3”, “Amazon Simple Email Service”, “Amazon Simple Notification Service”, “Amazon Simple Queue Service”, “Amazon Simple Storage Service”, “Amazon SimpleDB”, “Amazon SQS”, “Amazon Virtual Private Cloud”, “Amazon VPC”, “Amazon WorkDocs”, “Amazon WorkMail”, “AWS CloudFormation”, “AWS CloudHSM”, “AWS CloudTrail”, “AWS CodeBuild”, “AWS CodeCommit”, “AWS CodeDeploy”, “AWS CodePipeline”, “AWS CodeStar”, “AWS Direct Connect”, “AWS Glue”, “AWS Greengrass”, “AWS IoT Button”, “AWS Lambda”, “AWS Marketplace”, “AWS Migration Hub”, “AWS Premium Support”, “AWS Shield”, “AWS Snowball”, “AWS Snowball Edge”, “AWS Snowmobile”, “AWS Step Functions”, “AWS X-Ray”, “CloudFront”, “DevPay”, “DynamoDB”, “EC2”, “Elasticache”, “Mechanical Turk”, “SimpleDB”, “SQS”, and any other AWS Marks and Services made available from time to time.

PB_AWS_logo_RGB
PB_AWS_logo_RGB_WHT
3. Limited Permission. Provided that you are (a) an AWS developer in good standing with a current and valid account for use of the Services or (b) otherwise authorized by AWS in writing, and provided, further, that you comply at all times with the terms of both the Agreement and these Trademark Guidelines, we grant you a limited, non-exclusive, revocable, non-transferable permission, under our intellectual property rights in and to the AWS Marks, and only to the limited extent of our intellectual property rights in and to the AWS Marks, to use the AWS Marks for the following limited purpose, and only for such limited purpose: you may utilize the Logo or the appropriate form(s) of the “for” or equivalent naming convention or URL naming convention, as set forth in Section 9 below, to: (i) identify Your Content (as defined in the Agreement) as using the Services; or (ii) to identify software tools or applications that you create and distribute that are intended for use in connection with the Services. Without limitation of any provision in the Agreement, you acknowledge that any use that you elect to make of the AWS Marks, even if permitted hereunder, is at your sole risk and that we shall have no liability or responsibility in connection therewith. Your limited permission to use the AWS Marks is a limited permission and you may not use the AWS Marks for any other purpose. You may not transfer, assign or sublicense your limited permission to use the AWS Marks to any other person or entity. Your use of the AWS Marks shall comply with: (i) the most up-to-date versions of the Agreement and these Trademark Guidelines; and (ii) any other terms, conditions or policies that we may issue from time to time to govern use of the AWS Marks. Your limited permission to use the AWS Marks hereunder shall automatically terminate and you must immediately stop using the AWS Marks if at any time: (i) the Agreement is terminated; (ii) Your Content no longer uses any of the Services, or your software product cannot be used with any of the Services, as applicable; or (iii) you cease to be a registered AWS developer.

4. Modification and Termination. You understand and agree that, without prior notice to you and at our sole discretion: (i) we may modify these Trademark Guidelines at any time; (ii) we may modify or terminate your limited permission to use the AWS Marks, at any time in our sole discretion, for any reason or for no reason at all; and (iii) we reserve the right to take any and all actions including, without limitation, legal proceedings, against any use of the AWS Marks that does not comply with the terms of the Agreement or these Trademark Guidelines.

5. No Affiliation or Endorsement. You will not display the AWS Marks in any manner that implies that you are related to, affiliated with, sponsored or endorsed by us, or in a manner that could reasonably be interpreted to suggest that Your Content, web site, product or service, has been authored or edited by us, or represents our views or opinions.

6. No Disparagement. You may only use the AWS Marks in a manner designed to maintain the highest standard, quality and reputation that is associated with the AWS Marks and you will not use the AWS Marks to disparage us or our products or services.

7. No Dominant Display; AWS Mark Differentiation. You may not display any AWS Mark as the largest or most prominent trademark in any materials (including, without limitation, any web site or product literature) associated with Your Content, software tool or other software application. When using any AWS Mark (other than the Logo, with respect to which the formatting requirements are set forth in Section 8 below, or in a URL), you must distinguish the AWS Mark from the name of Your Content and/or other surrounding text by capitalizing the first letter of the AWS Mark, capitalizing or italicizing the entire AWS Mark, placing the AWS Mark in quotes, or using a different style or color of font for the AWS Mark.

8. Formatting Requirements with Respect to the “Powered by AWS” Logo.
a. No Modification. We will make the Logo image available to you from the co-marketing page in the AWS Site located at http://aws.amazon.com/co-marketing. You may not remove, distort or modify any element of the Logo; provided however, you may transform the file format itself, for ease of use.

b. Color. The Logo may be represented in the following formats: FULL COLOR (i) light backgrounds—squid ink type with Amazon Orange smile; (ii) dark backgrounds— white type with Amazon Orange smile; or for single-color applications; SINGLE COLOR (iii) light backgrounds—Squid Ink type with Squid Ink smile (preferred); GRAYSCALE (iv) light backgrounds—Black type with Black smile; (v) dark backgrounds— white type with white smile. No alternate color representation or combination will be acceptable.

When in FULL COLOR:
The graphic element (smile) must be in Amazon Orange
HEX: #FF9900
RGB: 255—153—0
CMYK: 0—45—95—0
PMS: COATED 1375 C
UNCOATED 137 U

The type must be in Squid Ink
HEX: #232F3E
RGB: 35—47—62
CMYK: 53—36—0—86
PMS: COATED 432 C

When in SINGLE COLOR:

Light Backgrounds: The graphic element (smile) and type must be in Squid Ink
HEX: #232F3E
RGB: 35—47—62
CMYK: 53—36—0—86
PMS: COATED 432 C

When in GRAYSCALE:

Light Backgrounds: The graphic element (smile) and type must be in Black
HEX: #000000
RGB: 0—0—0
CMYK: 0—0—0—100
PMS: Black

Dark Backgrounds: The graphic element (smile) and type must be in White
HEX: #FFFFFF
RGB: 255—255—255
CMYK: 0—0—0—0
PMS: White

c. Spacing. The Logo must appear by itself, with reasonable spacing (at least the height of the “Powered by AWS” logo) between each side of the “Powered by AWS” logo and other graphic or textual elements.

d. Size. The Logo, as shown in Section 2 of these Trademark Guidelines, indicates the minimum size at which you may display it to ensure that the type and trademark notations are legible. The minimum size for the “Powered by AWS” logo shall be: Inches: 2.625” x 1” Pixels: 190 × 70, or Millimeters: 67mm × 24mm.

9. Permissible Uses of the AWS Marks. Except for the Logo (with respect to which the formatting requirements are set forth above), you may only use the AWS Marks: (i) in a relational phrase using “for” or one of the limited number of equivalent naming conventions, as set forth below; or (ii) to the right of the top level domain name in a URL in the format set forth below.

• Relational Phrases.
Example of Permissible Use:
“Application” for EC2
Equivalents:
You may replace “for” in the example above with any of the following, so long as the term you use is accurate when used with the AWS Marks you use: “for use with”; “with”; “compatible with”; “works with”; “powered by”; “built on”; “built with”; “developed on”; “developed with.”
You may replace “EC2” in the examples above with any of the AWS Marks, so long as your usage of the AWS Marks is accurate.

• URLs.
Example of Permissible Use:
www.applicationdomain.com/aws
Equivalents:
You may replace “aws” in the example above with any of the AWS Marks, so long as your usage of the AWS Marks is accurate.

10. Hyperlinking. You shall link each use of the AWS Marks directly to the following URL, wherever technically feasible: http://aws.amazon.com. You may, alternatively, link to an AWS detail page for a Service used by Your Content, and if you do so, you must link to the primary URL for the Service (e.g., http://aws.amazon.com/[name of Service]. For example, the primary URL for Amazon Elastic Compute Cloud (Amazon EC2) is http://aws.amazon.com/ec2. You may open the URL in a new browser window. You may not link the AWS Marks to any web site other than the primary URL for the applicable Service. You may not frame or mirror any of our web site pages.

11. No Combination. You may not hyphenate, combine or abbreviate the AWS Marks. You shall not incorporate the AWS Marks into the name of your organization, or your services, products, trademark or logos. The foregoing prohibition includes the use of the AWS Marks in the name of any application, service or product or in a URL to the left of the top-level domain name (e.g., ”.com”, ”.net”, ”.uk”, etc.). For example, URLs such as “alexaaa.mydomain.com”, “mymechanicalturk.net” or “EC2plus2.com” are expressly prohibited.

12. Adwords. AWS and its affiliates are unable to provide a blanket authorization for use of our trademarks via the Google Adwords program. As long as your situation fits within Google’s “Reseller and information site policy”, you should be able to use the AWS trademarks in a fair use way within your adcopy without our formal authorization.

13. Attribution. You must include the following statement in any materials that include the AWS Marks: “Amazon Web Services, the “Powered by AWS” logo, [and name any other AWS Marks used in such materials] are trademarks of Amazon.com, Inc. or its affiliates in the United States and/or other countries."

14. No Misleading Use. You may not display the AWS Marks in any manner that is misleading, unfair, defamatory, infringing, libelous, disparaging, obscene or otherwise objectionable as determined by us in our sole discretion.

15. Trade Dress. You may not imitate the trade dress or “look and feel” of any of our web sites or pages contained in any of our web sites, including without limitation, the branding, color combinations, fonts, graphic designs, product icons or other elements associated with us.

16. Compliance with Law; Appropriate Activities. You may not use the AWS Marks in any manner that violates any United States or foreign, federal, state, provincial, municipal, local or other, law or regulation. Without limiting the foregoing, or any provision in the Agreement, you may not display any AWS Mark on your site if your site contains or displays adult content or promotes illegal activities, gambling, or the sale of tobacco or alcohol to persons under twenty-one (21) years of age.

17. Reservation of Rights. Except for the limited permission specified in Section 3 above, nothing in the Agreement or these Trademark Guidelines shall grant or be deemed to grant you any right, license, title or interest in or to any AWS Mark or any of our or our affiliates’ other trademarks, service marks, trade names, logos, product names, service names, legends, other designations, or abbreviations of any of the foregoing. You acknowledge and agree that we and our affiliates retain any and all intellectual property and other proprietary rights in and to the AWS Marks. All use by you of the AWS Marks including any goodwill associated therewith, shall inure to the benefit of Amazon.

18. No Challenges. You agree that you will not, at any time, challenge or encourage, assist or otherwise induce third parties to challenge the AWS Marks (except to the extent such restriction is prohibited by law) or our registration thereof, nor shall you attempt to register any trademarks, service marks, trade names, logos, product names, service names, legends, domain names, other designations, or abbreviations of any of the foregoing, or other distinctive brand features that are confusingly similar in any way (including, but not limited to, sound, appearance and spelling) to the AWS Marks.

19. Contact Information. If you have questions regarding your obligations under these Trademark Guidelines or questions about any AWS Mark, please contact or write to us at: Amazon.com, Inc., Attention: Trademarks, PO Box 81226, Seattle, WA 98108-1226.
Binary file added app/images/athena-logo.png
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
4 changes: 4 additions & 0 deletions backend/persistent/datastores/Datastores.js
Expand Up @@ -7,6 +7,7 @@ import * as ApacheLivy from './livy';
import * as ApacheImpala from './impala';
import * as DataWorld from './dataworld';
import * as DatastoreMock from './datastoremock';
import * as Athena from './athena';

const CSV = require('./csv');

Expand Down Expand Up @@ -36,6 +37,7 @@ function getDatastoreClient(connection) {
}

const {dialect} = connection;

if (dialect === 'elasticsearch') {
return Elasticsearch;
} else if (dialect === 's3') {
Expand All @@ -52,6 +54,8 @@ function getDatastoreClient(connection) {
return IbmDb2;
} else if (dialect === 'data.world') {
return DataWorld;
} else if (dialect === 'athena') {
return Athena;
}
return Sql;
}
Expand Down

0 comments on commit a2fe8bd

Please sign in to comment.