This repository has been archived by the owner on Jan 12, 2021. It is now read-only.
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Signed-off-by: mpeter28 <marciepeters28@yahoo.com>
- Loading branch information
Showing
36 changed files
with
2,061 additions
and
0 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,31 @@ | ||
Contributions require sign-off. The sign-off is required for all patch or pull requests, which certifies the following agreement given below. | ||
|
||
Contributor Agreement | ||
--------------------- | ||
|
||
By making a contribution to this project, I certify that: | ||
|
||
(a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or | ||
|
||
(b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or | ||
|
||
(c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. | ||
|
||
(d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. | ||
|
||
(e) I also agree to the following terms and conditions: | ||
|
||
(1) Grant of Copyright License. Subject to the terms and conditions of this agreement, You hereby grant to the maintainer and to recipients of software distributed by the maintainer a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute your contributions and such derivative works. | ||
|
||
(2) Grant of Patent License. Subject to the terms and conditions of this agreement, You hereby grant to the maintainer and to recipients of software distributed by the maintainer a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the work, where such license applies only to those patent claims licensable by you that are necessarily infringed by your contribution(s) alone or by combination of your contribution(s) with the work to which such contribution(s) was submitted. If any entity institutes patent litigation against you or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your contribution, or the work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this agreement for that contribution or work shall terminate as of the date such litigation is filed. | ||
|
||
Committing | ||
---------- | ||
|
||
Add a line stating | ||
|
||
Signed-off-by: Random J Developer <random@developer.example.org> | ||
|
||
When committing using the command line you can sign off using the --signoff or -s flag. This adds a Signed-off-by line by the committer at the end of the commit log message. | ||
|
||
git commit -s -m "Commit message" |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
# HiveQLUnit # | ||
|
||
HiveQLUnit is a unit testing framework for Hive HQL scripts. HiveQLUnit is low installation, managing the testing Hive Server for the user, and fully integrates with the JUnit unit testing framework. | ||
|
||
## License ## | ||
|
||
The HiveQLUnit project is licensed under Apache License Version 2.0 | ||
|
||
## User Guides ## | ||
|
||
* [Making an HQL Unit Project](userguides/MakingAnHiveQLUnitProject.md) | ||
* [Running HQL Unit Tests](userguides/RunningHiveQLUnitTests.md) | ||
* [Writing HQL Unit Tests](userguides/WritingHiveQLUnitTests.md) | ||
|
||
## Project Site ## | ||
|
||
http://finraos.github.io/HiveQLUnit/ | ||
|
||
## Current Release ## | ||
|
||
The latest release version is 1.0. | ||
|
||
<dependency> | ||
<groupId>org.finra.hiveqlunit</groupId> | ||
<artifactId>hiveQLUnit</artifactId> | ||
<version>1.0</version> | ||
</dependency> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,131 @@ | ||
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" | ||
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"> | ||
|
||
<modelVersion>4.0.0</modelVersion> | ||
<groupId>org.finra.hiveqlunit</groupId> | ||
<artifactId>hiveQLUnit</artifactId> | ||
<version>1.0</version> | ||
<packaging>jar</packaging> | ||
|
||
<name>HiveQLUnit</name> | ||
<description>Built on top of spark, HiveQLUnit is a unit testing framework for Hive HQL scripts</description> | ||
<licenses> | ||
<license> | ||
<name>The Apache Software License, Version 2.0</name> | ||
<url>http://www.apache.org/licenses/LICENSE-2.0.txt</url> | ||
</license> | ||
</licenses> | ||
<url>http://github.com/FINRAOS/HiveQLUnit</url> | ||
<scm> | ||
<connection>scm:git:http://github.com/FINRAOS/HiveQLUnit</connection> | ||
<developerConnection>scm:git:http://github.com/FINRAOS/HiveQLUnit</developerConnection> | ||
<url>http://github.com/FINRAOS/HiveQLUnit</url> | ||
</scm> | ||
<developers> | ||
<developer> | ||
<id>mpeter28</id> | ||
<name>Marcie Peters</name> | ||
<email>marciepeters28@yahoo.com</email> | ||
</developer> | ||
</developers> | ||
|
||
<properties> | ||
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> | ||
</properties> | ||
|
||
<build> | ||
<plugins> | ||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-compiler-plugin</artifactId> | ||
<version>2.1</version> | ||
<configuration> | ||
<source>1.7</source> | ||
<target>1.7</target> | ||
</configuration> | ||
</plugin> | ||
|
||
<plugin> | ||
<groupId>org.jacoco</groupId> | ||
<artifactId>jacoco-maven-plugin</artifactId> | ||
<version>0.6.3.201306030806</version> | ||
<executions> | ||
<execution> | ||
<id>jacoco-initialize</id> | ||
<goals> | ||
<goal>prepare-agent</goal> | ||
</goals> | ||
</execution> | ||
<execution> | ||
<id>jacoco-site</id> | ||
<phase>package</phase> | ||
<goals> | ||
<goal>report</goal> | ||
</goals> | ||
</execution> | ||
</executions> | ||
</plugin> | ||
|
||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-source-plugin</artifactId> | ||
<version>2.2.1</version> | ||
<executions> | ||
<execution> | ||
<id>attach-sources</id> | ||
<goals> | ||
<goal>jar</goal> | ||
</goals> | ||
</execution> | ||
</executions> | ||
</plugin> | ||
|
||
<plugin> | ||
<groupId>org.apache.maven.plugins</groupId> | ||
<artifactId>maven-javadoc-plugin</artifactId> | ||
<version>2.9.1</version> | ||
<executions> | ||
<execution> | ||
<id>attach-javadocs</id> | ||
<goals> | ||
<goal>jar</goal> | ||
</goals> | ||
</execution> | ||
</executions> | ||
</plugin> | ||
</plugins> | ||
</build> | ||
|
||
<dependencies> | ||
<dependency> | ||
<groupId>junit</groupId> | ||
<artifactId>junit</artifactId> | ||
<version>4.9</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.apache.commons</groupId> | ||
<artifactId>commons-io</artifactId> | ||
<version>1.3.2</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.apache.spark</groupId> | ||
<artifactId>spark-core_2.10</artifactId> | ||
<version>1.4.1</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.apache.spark</groupId> | ||
<artifactId>spark-sql_2.10</artifactId> | ||
<version>1.4.1</version> | ||
</dependency> | ||
|
||
<dependency> | ||
<groupId>org.apache.spark</groupId> | ||
<artifactId>spark-hive_2.10</artifactId> | ||
<version>1.4.1</version> | ||
</dependency> | ||
</dependencies> | ||
|
||
</project> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,66 @@ | ||
/* | ||
* Copyright 2016 HiveQLUnit Contributors | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.finra.hiveqlunit; | ||
|
||
import junit.framework.Assert; | ||
import org.apache.spark.sql.Row; | ||
import org.finra.hiveqlunit.resources.TextResource; | ||
import org.finra.hiveqlunit.rules.TestDataLoader; | ||
import org.finra.hiveqlunit.rules.TestHiveServer; | ||
import org.finra.hiveqlunit.script.MultiExpressionScript; | ||
import org.junit.ClassRule; | ||
import org.junit.Rule; | ||
|
||
public class HiveQLUnitTest { | ||
|
||
@ClassRule | ||
public static TestHiveServer hiveServer = new TestHiveServer(); | ||
|
||
@Rule | ||
public static TestDataLoader loader = new TestDataLoader(hiveServer); | ||
|
||
public static void createTable(TextResource tableDDL) { | ||
new MultiExpressionScript(tableDDL).runScript(hiveServer.getHiveContext()); | ||
} | ||
|
||
public static void loadTableData(String tableName, TextResource tableData) { | ||
loader.loadDataIntoTable(tableName, tableData); | ||
} | ||
|
||
public static void loadTableData(String tableName, TextResource tableData, String partitionInfo) { | ||
loader.loadDataIntoTable(tableName, tableData, partitionInfo); | ||
} | ||
|
||
public static void runScript(TextResource script) { | ||
new MultiExpressionScript(script).runScript(hiveServer.getHiveContext()); | ||
} | ||
|
||
public static void diffActualAndExpected(TextResource diffScript) { | ||
Row[] rows = new MultiExpressionScript(diffScript).runScriptReturnResults(hiveServer.getHiveContext()); | ||
if (rows.length != 0) { | ||
Assert.fail(); | ||
} | ||
} | ||
|
||
public static void diffActualAndExpected(TextResource diffScript, TextResource tearDownScript) { | ||
try { | ||
diffActualAndExpected(diffScript); | ||
} finally { | ||
new MultiExpressionScript(tearDownScript).runScript(hiveServer.getHiveContext()); | ||
} | ||
} | ||
} |
56 changes: 56 additions & 0 deletions
56
src/main/java/org/finra/hiveqlunit/resources/InputStreamResource.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,56 @@ | ||
/* | ||
* Copyright 2016 HiveQLUnit Contributors | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.finra.hiveqlunit.resources; | ||
|
||
import org.apache.commons.io.IOUtils; | ||
|
||
import java.io.IOException; | ||
import java.io.InputStream; | ||
import java.io.StringWriter; | ||
|
||
/** | ||
* Provides an abstract TextResource for building TextResource classes that access InputStream accessible resources | ||
* | ||
* InputStreamResource handles the utility of reading out the content of an InputStream into a String | ||
* | ||
* Child classes provide the InputStream to read from | ||
*/ | ||
public abstract class InputStreamResource implements TextResource { | ||
|
||
/** | ||
* Provides an InputStream with the text content of the TextResource | ||
* | ||
* @return an InputStream to read the text content of the TextResource from | ||
* @throws IOException if the InputStream can not be constructed | ||
*/ | ||
public abstract InputStream resourceStream() throws IOException; | ||
|
||
/** | ||
* Provides the text content of the TextResource by reading out the content of the resource's InputStream | ||
* @return the text content of the TextResource, as contained in its provided InputStream | ||
*/ | ||
@Override | ||
public String resourceText() { | ||
try { | ||
StringWriter writer = new StringWriter(); | ||
IOUtils.copy(resourceStream(), writer, "UTF-8"); | ||
return writer.getBuffer().toString(); | ||
} catch (IOException e) { | ||
throw new RuntimeException("Failure to load resource"); | ||
} | ||
} | ||
} |
49 changes: 49 additions & 0 deletions
49
src/main/java/org/finra/hiveqlunit/resources/LocalFileResource.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,49 @@ | ||
/* | ||
* Copyright 2016 HiveQLUnit Contributors | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
package org.finra.hiveqlunit.resources; | ||
|
||
import java.io.File; | ||
import java.io.FileInputStream; | ||
import java.io.IOException; | ||
import java.io.InputStream; | ||
|
||
/** | ||
* Handles access to text resources contained in files on the local file system | ||
*/ | ||
public class LocalFileResource extends InputStreamResource { | ||
|
||
private String filePath; | ||
|
||
/** | ||
* Constructs a TextResource that reads out the content of a file on the local file system | ||
* | ||
* @param filePath the file system path of the file | ||
*/ | ||
public LocalFileResource(String filePath) { | ||
this.filePath = filePath; | ||
} | ||
|
||
/** | ||
* Files ion the local file system are easily accessed as InputStreams | ||
* | ||
* @return an InputStream with the contents of the local file | ||
*/ | ||
@Override | ||
public InputStream resourceStream() throws IOException{ | ||
return new FileInputStream(new File(filePath)); | ||
} | ||
} |
Oops, something went wrong.