diff --git a/DCO b/DCO new file mode 100644 index 0000000..6e1467d --- /dev/null +++ b/DCO @@ -0,0 +1,31 @@ +Contributions require sign-off. The sign-off is required for all patch or pull requests, which certifies the following agreement given below. + +Contributor Agreement +--------------------- + + By making a contribution to this project, I certify that: + + (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or + + (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or + + (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. + + (d) I understand and agree that this project and the contribution are public and that a record of the contribution (including all personal information I submit with it, including my sign-off) is maintained indefinitely and may be redistributed consistent with this project or the open source license(s) involved. + + (e) I also agree to the following terms and conditions: + + (1) Grant of Copyright License. Subject to the terms and conditions of this agreement, You hereby grant to the maintainer and to recipients of software distributed by the maintainer a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare derivative works of, publicly display, publicly perform, sublicense, and distribute your contributions and such derivative works. + + (2) Grant of Patent License. Subject to the terms and conditions of this agreement, You hereby grant to the maintainer and to recipients of software distributed by the maintainer a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the work, where such license applies only to those patent claims licensable by you that are necessarily infringed by your contribution(s) alone or by combination of your contribution(s) with the work to which such contribution(s) was submitted. If any entity institutes patent litigation against you or any other entity (including a cross-claim or counterclaim in a lawsuit) alleging that your contribution, or the work to which you have contributed, constitutes direct or contributory patent infringement, then any patent licenses granted to that entity under this agreement for that contribution or work shall terminate as of the date such litigation is filed. + +Committing +---------- + +Add a line stating + + Signed-off-by: Random J Developer + +When committing using the command line you can sign off using the --signoff or -s flag. This adds a Signed-off-by line by the committer at the end of the commit log message. + + git commit -s -m "Commit message" diff --git a/README.md b/README.md new file mode 100644 index 0000000..b7b32c7 --- /dev/null +++ b/README.md @@ -0,0 +1,27 @@ +# HiveQLUnit # + +HiveQLUnit is a unit testing framework for Hive HQL scripts. HiveQLUnit is low installation, managing the testing Hive Server for the user, and fully integrates with the JUnit unit testing framework. + +## License ## + +The HiveQLUnit project is licensed under Apache License Version 2.0 + +## User Guides ## + +* [Making an HQL Unit Project](userguides/MakingAnHiveQLUnitProject.md) +* [Running HQL Unit Tests](userguides/RunningHiveQLUnitTests.md) +* [Writing HQL Unit Tests](userguides/WritingHiveQLUnitTests.md) + +## Project Site ## + +http://finraos.github.io/HiveQLUnit/ + +## Current Release ## + +The latest release version is 1.0. + + + org.finra.hiveqlunit + hiveQLUnit + 1.0 + \ No newline at end of file diff --git a/pom.xml b/pom.xml new file mode 100644 index 0000000..6a39a3d --- /dev/null +++ b/pom.xml @@ -0,0 +1,131 @@ + + + 4.0.0 + org.finra.hiveqlunit + hiveQLUnit + 1.0 + jar + + HiveQLUnit + Built on top of spark, HiveQLUnit is a unit testing framework for Hive HQL scripts + + + The Apache Software License, Version 2.0 + http://www.apache.org/licenses/LICENSE-2.0.txt + + + http://github.com/FINRAOS/HiveQLUnit + + scm:git:http://github.com/FINRAOS/HiveQLUnit + scm:git:http://github.com/FINRAOS/HiveQLUnit + http://github.com/FINRAOS/HiveQLUnit + + + + mpeter28 + Marcie Peters + marciepeters28@yahoo.com + + + + + UTF-8 + + + + + + org.apache.maven.plugins + maven-compiler-plugin + 2.1 + + 1.7 + 1.7 + + + + + org.jacoco + jacoco-maven-plugin + 0.6.3.201306030806 + + + jacoco-initialize + + prepare-agent + + + + jacoco-site + package + + report + + + + + + + org.apache.maven.plugins + maven-source-plugin + 2.2.1 + + + attach-sources + + jar + + + + + + + org.apache.maven.plugins + maven-javadoc-plugin + 2.9.1 + + + attach-javadocs + + jar + + + + + + + + + + junit + junit + 4.9 + + + + org.apache.commons + commons-io + 1.3.2 + + + + org.apache.spark + spark-core_2.10 + 1.4.1 + + + + org.apache.spark + spark-sql_2.10 + 1.4.1 + + + + org.apache.spark + spark-hive_2.10 + 1.4.1 + + + + diff --git a/src/main/java/org/finra/hiveqlunit/HiveQLUnitTest.java b/src/main/java/org/finra/hiveqlunit/HiveQLUnitTest.java new file mode 100644 index 0000000..05f5fb6 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/HiveQLUnitTest.java @@ -0,0 +1,66 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit; + +import junit.framework.Assert; +import org.apache.spark.sql.Row; +import org.finra.hiveqlunit.resources.TextResource; +import org.finra.hiveqlunit.rules.TestDataLoader; +import org.finra.hiveqlunit.rules.TestHiveServer; +import org.finra.hiveqlunit.script.MultiExpressionScript; +import org.junit.ClassRule; +import org.junit.Rule; + +public class HiveQLUnitTest { + + @ClassRule + public static TestHiveServer hiveServer = new TestHiveServer(); + + @Rule + public static TestDataLoader loader = new TestDataLoader(hiveServer); + + public static void createTable(TextResource tableDDL) { + new MultiExpressionScript(tableDDL).runScript(hiveServer.getHiveContext()); + } + + public static void loadTableData(String tableName, TextResource tableData) { + loader.loadDataIntoTable(tableName, tableData); + } + + public static void loadTableData(String tableName, TextResource tableData, String partitionInfo) { + loader.loadDataIntoTable(tableName, tableData, partitionInfo); + } + + public static void runScript(TextResource script) { + new MultiExpressionScript(script).runScript(hiveServer.getHiveContext()); + } + + public static void diffActualAndExpected(TextResource diffScript) { + Row[] rows = new MultiExpressionScript(diffScript).runScriptReturnResults(hiveServer.getHiveContext()); + if (rows.length != 0) { + Assert.fail(); + } + } + + public static void diffActualAndExpected(TextResource diffScript, TextResource tearDownScript) { + try { + diffActualAndExpected(diffScript); + } finally { + new MultiExpressionScript(tearDownScript).runScript(hiveServer.getHiveContext()); + } + } +} diff --git a/src/main/java/org/finra/hiveqlunit/resources/InputStreamResource.java b/src/main/java/org/finra/hiveqlunit/resources/InputStreamResource.java new file mode 100644 index 0000000..0fddd1c --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/resources/InputStreamResource.java @@ -0,0 +1,56 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +import org.apache.commons.io.IOUtils; + +import java.io.IOException; +import java.io.InputStream; +import java.io.StringWriter; + +/** + * Provides an abstract TextResource for building TextResource classes that access InputStream accessible resources + * + * InputStreamResource handles the utility of reading out the content of an InputStream into a String + * + * Child classes provide the InputStream to read from + */ +public abstract class InputStreamResource implements TextResource { + + /** + * Provides an InputStream with the text content of the TextResource + * + * @return an InputStream to read the text content of the TextResource from + * @throws IOException if the InputStream can not be constructed + */ + public abstract InputStream resourceStream() throws IOException; + + /** + * Provides the text content of the TextResource by reading out the content of the resource's InputStream + * @return the text content of the TextResource, as contained in its provided InputStream + */ + @Override + public String resourceText() { + try { + StringWriter writer = new StringWriter(); + IOUtils.copy(resourceStream(), writer, "UTF-8"); + return writer.getBuffer().toString(); + } catch (IOException e) { + throw new RuntimeException("Failure to load resource"); + } + } +} diff --git a/src/main/java/org/finra/hiveqlunit/resources/LocalFileResource.java b/src/main/java/org/finra/hiveqlunit/resources/LocalFileResource.java new file mode 100644 index 0000000..7ca4559 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/resources/LocalFileResource.java @@ -0,0 +1,49 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +import java.io.File; +import java.io.FileInputStream; +import java.io.IOException; +import java.io.InputStream; + +/** + * Handles access to text resources contained in files on the local file system + */ +public class LocalFileResource extends InputStreamResource { + + private String filePath; + + /** + * Constructs a TextResource that reads out the content of a file on the local file system + * + * @param filePath the file system path of the file + */ + public LocalFileResource(String filePath) { + this.filePath = filePath; + } + + /** + * Files ion the local file system are easily accessed as InputStreams + * + * @return an InputStream with the contents of the local file + */ + @Override + public InputStream resourceStream() throws IOException{ + return new FileInputStream(new File(filePath)); + } +} diff --git a/src/main/java/org/finra/hiveqlunit/resources/ResourceFolderResource.java b/src/main/java/org/finra/hiveqlunit/resources/ResourceFolderResource.java new file mode 100644 index 0000000..711e4bd --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/resources/ResourceFolderResource.java @@ -0,0 +1,48 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +import java.io.InputStream; + +/** + * Jars can have files packaged into the resources folder of the jar + * + * This TextResource handles access to said resource folder files + */ +public class ResourceFolderResource extends InputStreamResource { + + private String resourcePath; + + /** + * Constructs a TextResource that reads out the content of a file in the resources folder of a jar + * + * @param resourcePath the full path of the resource folder file, ie '/foo/bar/file.dat' + */ + public ResourceFolderResource(String resourcePath) { + this.resourcePath = resourcePath; + } + + /** + * Files in the resource folder are easily accessed as InputStreams + * + * @return an InputStream with the contents of the resource folder file + */ + @Override + public InputStream resourceStream() { + return ResourceFolderResource.class.getResourceAsStream(resourcePath); + } +} diff --git a/src/main/java/org/finra/hiveqlunit/resources/TextLiteralResource.java b/src/main/java/org/finra/hiveqlunit/resources/TextLiteralResource.java new file mode 100644 index 0000000..452d11e --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/resources/TextLiteralResource.java @@ -0,0 +1,44 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +/** + * The most basic TextResource - wraps text literals or Java strings as a TextResource + */ +public class TextLiteralResource implements TextResource { + + private String resourceText; + + /** + * Constructs a TextResource that wraps a string value, which is the resources' text content + * + * @param resourceText the resources' text content + */ + public TextLiteralResource(String resourceText) { + this.resourceText = resourceText; + } + + /** + * This TextResource wraps a string value, which is the resources' text content + * + * @return the resourceText param passed to the constructor + */ + @Override + public String resourceText() { + return resourceText; + } +} diff --git a/src/main/java/org/finra/hiveqlunit/resources/TextResource.java b/src/main/java/org/finra/hiveqlunit/resources/TextResource.java new file mode 100644 index 0000000..a39d548 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/resources/TextResource.java @@ -0,0 +1,32 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +/** + * Abstracts access to textual resources needed during testing, such as test data or hql scripts + * + * By using this interface, code that uses testing resources can be written agnostic to the origin or nature of the resource + */ +public interface TextResource { + + /** + * Provides the text content of the resource the TextResource object represents + * + * @return the text content of the resource the TextResource object represents + */ + public String resourceText(); +} diff --git a/src/main/java/org/finra/hiveqlunit/resources/manipulation/Dos2UnixResource.java b/src/main/java/org/finra/hiveqlunit/resources/manipulation/Dos2UnixResource.java new file mode 100644 index 0000000..165fca9 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/resources/manipulation/Dos2UnixResource.java @@ -0,0 +1,50 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources.manipulation; + +import org.finra.hiveqlunit.resources.TextResource; + +/** + * A decorative TextResource which can wrap any TextResource + * + * Replaces \r\n line endings in the wrapped TextResource with \n line endings + * + * The wrapped resource is not actually altered, it only looks different to calling code + */ +public class Dos2UnixResource implements TextResource { + + private TextResource baseResource; + + /** + * Constructs a TextResource that disguises Windows line endings in a wrapped resource with Unix line endings + * + * @param baseResource the TextResource to wrap + */ + public Dos2UnixResource(TextResource baseResource) { + this.baseResource = baseResource; + } + + /** + * Reads the text content of the wrapped TextResource, then changes the line endings of what was read + * + * @return the text content of the wrapped TextResource, but with Unix line endings + */ + @Override + public String resourceText() { + return baseResource.resourceText().replaceAll("\r\n", "\n"); + } +} diff --git a/src/main/java/org/finra/hiveqlunit/resources/manipulation/SubstituteVariableResource.java b/src/main/java/org/finra/hiveqlunit/resources/manipulation/SubstituteVariableResource.java new file mode 100644 index 0000000..5f2fce3 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/resources/manipulation/SubstituteVariableResource.java @@ -0,0 +1,57 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources.manipulation; + +import org.finra.hiveqlunit.resources.TextResource; + +/** + * A decorative TextResource which can wrap any TextResource + * + * Substitutes variables in a TextResources content represented by ${variableName} with a desired value + * Replaces all such instances of ${variableName} with the desired value + * + * The wrapped resource is not actually altered, it only looks different to calling code + */ +public class SubstituteVariableResource implements TextResource { + + private String variableName; + private String replacementValue; + private TextResource baseResource; + + /** + * Constructs a TextResource that substitutes variables with values in a wrapped TextResource + * + * @param variableName the variable name, as seen within ${}, ie ${variableName} + * @param replacementValue the value to replace ${variableName} with + * @param baseResource the wrapped TextResource + */ + public SubstituteVariableResource(String variableName, String replacementValue, TextResource baseResource) { + this.variableName = variableName; + this.replacementValue = replacementValue; + this.baseResource = baseResource; + } + + /** + * Reads the text content of the wrapped TextResource, then substitutes variable instances with the correct value + * + * @return the text content of the wrapped TextResource, but with variable substitution + */ + @Override + public String resourceText() { + return baseResource.resourceText().replaceAll("\\$\\{" + variableName + "\\}", replacementValue); + } +} diff --git a/src/main/java/org/finra/hiveqlunit/rules/SetUpHql.java b/src/main/java/org/finra/hiveqlunit/rules/SetUpHql.java new file mode 100644 index 0000000..43b148c --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/rules/SetUpHql.java @@ -0,0 +1,95 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +import org.finra.hiveqlunit.script.HqlScript; +import org.junit.rules.TestRule; +import org.junit.runner.Description; +import org.junit.runners.model.Statement; + +/** + * A TestRule which executes hive scripts before the test is executed as part of a set up effort + * + * The order of evaluation amongst multiple SetUpHql rules is *NON DETERMINISTIC* + * If evaluation order matters, use the same rule to execute multiple set up commands as part of the same script + */ +public class SetUpHql implements TestRule { + + private TestHiveServer testingHiveServer; + private HqlScript setUpHql; + + /** + * Constructs a SetUpHql object + * + * @param testingHiveServer the TestHiveServer to run the set up on + * @param setUpHql a TextResource containing with the set up script to run + */ + public SetUpHql(TestHiveServer testingHiveServer, HqlScript setUpHql) { + this.testingHiveServer = testingHiveServer; + this.setUpHql = setUpHql; + } + + /** + * Wraps a given Statement with a RunSetUpHqlStatement instance + * + * When the nested Statement chain is evaluated, set up script is run, and then the wrapped Statement is evaluated + * + * @param statement the Statement to run the set up script before evaluation of + * @param description ignored + * @return the given Statement wrapped with a RunSetUpHqlStatement + */ + @Override + public Statement apply(Statement statement, Description description) { + return new RunSetUpHqlStatement(statement, testingHiveServer, setUpHql); + } + + /** + * A Statement that does the actual heavy lifting for SetUpHql + */ + public static class RunSetUpHqlStatement extends Statement { + + private Statement wrappedStatement; + private TestHiveServer testingHiveServer; + private HqlScript setUpHql; + + /** + * Constructs a RunSetUpHqlStatement + * + * When evaluated this Statement runs a hive script as set up code and then evaluates a wrapped Statement + * + * @param wrappedStatement the Statement to execute the set up script before evaluation of + * @param testingHiveServer the TestHiveServer to run the set up script on + * @param setUpHql a TextResource containing the hql script to perform set up with + */ + public RunSetUpHqlStatement(Statement wrappedStatement, TestHiveServer testingHiveServer, HqlScript setUpHql) { + this.wrappedStatement = wrappedStatement; + this.testingHiveServer = testingHiveServer; + this.setUpHql = setUpHql; + } + + /** + * Executes the set up hql script, and then evaluates the wrapped Statement + * + * @throws Throwable as required by the Statement class + */ + @Override + public void evaluate() throws Throwable { + setUpHql.runScript(testingHiveServer.getHiveContext()); + wrappedStatement.evaluate(); + } + } +} diff --git a/src/main/java/org/finra/hiveqlunit/rules/TearDownHql.java b/src/main/java/org/finra/hiveqlunit/rules/TearDownHql.java new file mode 100644 index 0000000..8ff28a7 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/rules/TearDownHql.java @@ -0,0 +1,102 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +import org.finra.hiveqlunit.script.HqlScript; +import org.junit.rules.TestRule; +import org.junit.runner.Description; +import org.junit.runners.model.Statement; + +/** + * A TestRule which executes hive scripts after the test is done as part of a clean up effort + * + * The order of evaluation amongst multiple TearDownHql rules is *NON DETERMINISTIC* + * If evaluation order matters, use the same rule to execute multiple tear down commands as part of the same script + * + * If the test method (annotated with @Test) fails, the tear down script will still be run + */ +public class TearDownHql implements TestRule { + + private TestHiveServer testingHiveServer; + private HqlScript tearDownHql; + + /** + * Constructs a TearDownHql object + * + * @param testingHiveServer the TestHiveServer to run the tear down script on + * @param tearDownHql a TextResource containing with the tear down script to run + */ + public TearDownHql(TestHiveServer testingHiveServer, HqlScript tearDownHql) { + this.testingHiveServer = testingHiveServer; + this.tearDownHql = tearDownHql; + } + + /** + * Wraps a given Statement with a RunTearDownHqlStatement instance + * + * When the nested Statement chain is evaluated, the tear down script will run after the wrapped statement is evaluated + * + * @param statement the Statement to run the tear down script after evaluation of + * @param description ignored + * @return the given Statement wrapped with a RunTearDownHqlStatement + */ + @Override + public Statement apply(Statement statement, Description description) { + return new RunTearDownHqlStatement(statement, testingHiveServer, tearDownHql); + } + + /** + * A Statement that does the actual heavy lifting for TearDownHql + */ + public static class RunTearDownHqlStatement extends Statement { + + private Statement wrappedStatement; + private TestHiveServer testingHiveServer; + private HqlScript tearDownHql; + + /** + * Constructs a RunTearDownHqlStatement + * + * When evaluated this Statement evaluates a wrapped statement and then runs an hql script as testing tear down + * + * @param wrappedStatement the Statement to execute the tear down script after evaluation of + * @param testingHiveServer the TestHiveServer to run the tear down script on + * @param tearDownHql a TextResource containing the hql script to perform tear down with + */ + public RunTearDownHqlStatement(Statement wrappedStatement, TestHiveServer testingHiveServer, HqlScript tearDownHql) { + this.wrappedStatement = wrappedStatement; + this.testingHiveServer = testingHiveServer; + this.tearDownHql = tearDownHql; + } + + /** + * Evaluates the wrapped statement, followed by executing the tear down script + * + * The script will always be executed even if a test fails or an error has been thrown + * + * @throws Throwable as required by the Statement class + */ + @Override + public void evaluate() throws Throwable { + try { + wrappedStatement.evaluate(); + } finally { + tearDownHql.runScript(testingHiveServer.getHiveContext()); + } + } + } +} \ No newline at end of file diff --git a/src/main/java/org/finra/hiveqlunit/rules/TestDataLoader.java b/src/main/java/org/finra/hiveqlunit/rules/TestDataLoader.java new file mode 100644 index 0000000..c7f8cd5 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/rules/TestDataLoader.java @@ -0,0 +1,100 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +import org.finra.hiveqlunit.resources.TextResource; +import org.junit.rules.TemporaryFolder; +import org.junit.rules.TestRule; +import org.junit.runner.Description; +import org.junit.runners.model.Statement; + +import java.io.File; +import java.io.IOException; +import java.io.PrintWriter; + +/** + * This TestRule class provides utility functions for extracting testing data from resources and loading it into Hive tables + */ +public class TestDataLoader implements TestRule { + + private TestHiveServer hiveServer; + private TemporaryFolder stagingLocation; + + /** + * Constructs a new TestDataLoader + * + * @param hiveServer the TestHiveServer to load data into + */ + public TestDataLoader(TestHiveServer hiveServer) { + this.hiveServer = hiveServer; + stagingLocation = new TemporaryFolder(); + } + + /** + * Uses a TemporaryFolder rule to process a given statement + * + * The TemporaryFolder will be used to construct staging files for loading data into Hive + * + * @param base the statement to apply a TemporaryFolder to + * @param description the description to apply a TemporaryFolder to + * @return the output of the TemporaryFolder rule + */ + @Override + public Statement apply(Statement base, Description description) { + return stagingLocation.apply(base, description); + } + + /** + * Loads data from a TextResource into a hive table + * + * A temporary file on the local file system is used as a staging ground for the data + * + * @param tableName the name of the table + * @param tableDataResource the resource to extract data from + */ + public void loadDataIntoTable(String tableName, TextResource tableDataResource) { + loadDataIntoTable(tableName, tableDataResource, ""); + } + + /** + * Loads data from a TextResource into a hive table + * + * A temporary file on the local file system is used as a staging ground for the data + * + * @param tableName the name of the table + * @param tableDataResource the resource to extract data from + * @param partitionInfo the optional partitioning commands at the end of a 'LOAD DATA' query + */ + public void loadDataIntoTable(String tableName, TextResource tableDataResource, String partitionInfo) { + try { + File stagingFile = stagingLocation.newFile(tableName + ".dat"); + String tableData = tableDataResource.resourceText(); + PrintWriter writer = new PrintWriter(stagingFile.getAbsoluteFile()); + writer.print(tableData); + writer.close(); + + hiveServer.getHiveContext().sql("LOAD DATA LOCAL INPATH '" + + stagingFile.getAbsolutePath().replace("\\", "/") + + "' INTO TABLE " + + tableName + + " " + + partitionInfo); + } catch (IOException e) { + throw new RuntimeException("Failure to load table data"); + } + } +} diff --git a/src/main/java/org/finra/hiveqlunit/rules/TestHiveServer.java b/src/main/java/org/finra/hiveqlunit/rules/TestHiveServer.java new file mode 100644 index 0000000..22bfe51 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/rules/TestHiveServer.java @@ -0,0 +1,108 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +import org.apache.spark.SparkConf; +import org.apache.spark.api.java.JavaSparkContext; +import org.apache.spark.sql.hive.HiveContext; +import org.junit.rules.TestRule; +import org.junit.runner.Description; +import org.junit.runners.model.Statement; + +/** + * TestHiveServer is a TestRule responsible for constructing a HiveContext for use in testing hql scripts + * + * *MUST* be used with the @ClassRule annotation - other HqlUnit provided TestRules require this rule to run first + * + * Many classes take a TestHiveServer as an input even though they just need the HiveContext. + * TestHiveServer serves as a passable reference to the HiveContext before the context has actually been made + */ +public class TestHiveServer implements TestRule { + + private ConstructHiveContextStatement constructContext; + + /** + * Wraps a given statement with a ConstructHiveContextStatement + * + * @param statement the base statement to be wrapped + * @param description ignored + * @return a ConstructHiveContextStatement instance which wraps the provided statement + */ + @Override + public Statement apply(Statement statement, Description description) { + constructContext = new ConstructHiveContextStatement(statement); + return constructContext; + } + + /** + * Provides access to the HiveContext produced by this TestRule + * + * @return the HiveContext produced by this TestRule + */ + public HiveContext getHiveContext() { + return constructContext.getHiveContext(); + } + + /** + * A Statement that performs most of the work for TestHiveServer + */ + public static class ConstructHiveContextStatement extends Statement { + + private static HiveContext hiveContextSingleton; + + private Statement wrappedStatement; + + /** + * Wraps a given Statement + * + * This Statement constructs the all important HiveContext, then evaluates the wrapped Statement + * + * @param wrappedStatement the statement to wrap, which will be evaluated after the HiveContext is made + */ + public ConstructHiveContextStatement(Statement wrappedStatement) { + this.wrappedStatement = wrappedStatement; + } + + /** + * Constructs the all important HiveContext, then evaluates the wrapped Statement + * + * Currently, the HiveContext is made as a singleton + * + * @throws Throwable as required by the Statement class + */ + @Override + public void evaluate() throws Throwable { + if (hiveContextSingleton == null) { + SparkConf sparkConf = new SparkConf().setAppName("Test HQL").setMaster("local[1]"); + JavaSparkContext sparkContext = new JavaSparkContext(sparkConf); + + hiveContextSingleton = new HiveContext(sparkContext.sc()); + } + + wrappedStatement.evaluate(); + } + + /** + * Provides access to the HiveContext produced by this Statement + * + * @return the HiveContext produced by this TestRule + */ + public HiveContext getHiveContext() { + return hiveContextSingleton; + } + } +} \ No newline at end of file diff --git a/src/main/java/org/finra/hiveqlunit/script/HqlScript.java b/src/main/java/org/finra/hiveqlunit/script/HqlScript.java new file mode 100644 index 0000000..ef03bd5 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/script/HqlScript.java @@ -0,0 +1,46 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.script; + +import org.apache.spark.sql.Row; +import org.apache.spark.sql.hive.HiveContext; + +/** + * Abstracts the execution of hql scripts or expressions + * + * How to handle comments in scripts, or split scripts into runnable expressions is handled here + * + * This is separate from acquiring the text representing the hql script or expression, TextResources better handle that + */ +public interface HqlScript { + + /** + * Runs the hql script or expressions represented by this HqlScript using a HiveContext + * + * @param hqlContext an HqlContext, as provided by spark through the TestHiveServer TestRule, used to run hql expressions + */ + public void runScript(HiveContext hqlContext); + + /** + * Runs the hql script or expressions represented by this HqlScript using a HiveContext, returning a results set from the script + * + * @param hqlContext an HqlContext, as provided by spark through the TestHiveServer TestRule, used to run hql expressions + * @return a result set of Rows produced by running the hql script or expressions represented by this HqlScript + */ + public Row[] runScriptReturnResults(HiveContext hqlContext); + +} diff --git a/src/main/java/org/finra/hiveqlunit/script/MultiExpressionScript.java b/src/main/java/org/finra/hiveqlunit/script/MultiExpressionScript.java new file mode 100644 index 0000000..3d10d3f --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/script/MultiExpressionScript.java @@ -0,0 +1,71 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.script; + +import org.apache.spark.sql.Row; +import org.apache.spark.sql.hive.HiveContext; +import org.finra.hiveqlunit.resources.TextResource; + +/** + * Runs an hql script containing multiple expressions, using the ScriptSplitter utility to derive individual expressions + */ +public class MultiExpressionScript implements HqlScript { + + private String script; + + /** + * Constructs a MultiExpressionScript + * + * @param expressionResource a TextResource containing an hql script for MultiExpressionScript to run + */ + public MultiExpressionScript(TextResource expressionResource) { + script = expressionResource.resourceText(); + } + + /** + * Splits the bundled hql script into multiple expressions using ScriptSlitter utility class + * + * Each expression is run on the provided HiveContext + * + * @param hqlContext an HqlContext, as provided by spark through the TestHiveServer TestRule, used to run hql expressions + */ + @Override + public void runScript(HiveContext hqlContext) { + String[] expressions = ScriptSplitter.splitScriptIntoExpressions(script); + for (String expression : expressions) { + hqlContext.sql(expression); + } + } + + /** + * Splits the bundled hql script into multiple expressions using ScriptSlitter utility class + * + * Each expression is run on the provided HiveContext + * + * @param hqlContext an HqlContext, as provided by spark through the TestHiveServer TestRule, used to run hql expressions + * @return the row results acquired from the last executed expression + */ + @Override + public Row[] runScriptReturnResults(HiveContext hqlContext) { + String[] expressions = ScriptSplitter.splitScriptIntoExpressions(script); + for (int i = 0; i < expressions.length - 1; i++) { + String expression = expressions[i]; + hqlContext.sql(expression); + } + return hqlContext.sql(expressions[expressions.length - 1]).collect(); + } +} diff --git a/src/main/java/org/finra/hiveqlunit/script/ScriptSplitter.java b/src/main/java/org/finra/hiveqlunit/script/ScriptSplitter.java new file mode 100644 index 0000000..710f725 --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/script/ScriptSplitter.java @@ -0,0 +1,45 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.script; + +/** + * A static only utility class with functionality to split hql scripts into multiple expressions + * + * Needs some work, the quality of the split it not great; many legal scripts might be parsed wrong + */ +public class ScriptSplitter { + + /** + * Takes an hql script represented as a String and splits it into multiple hql expressions + * + * Expressions are assumed to end with a semi colon followed by a new line (unix style \n or windows style \r\n) + * + * Comments are removed, but only if they start at the beginning of a line + * + * @param script the hql script to split into an expression + * @return an array of String hql expressions + */ + public static String[] splitScriptIntoExpressions(String script) { + String[] expressions = script.split(";\n|;\r\n|;$"); + + for (int i = 0; i < expressions.length; i++) { + expressions[i] = expressions[i].replaceAll("--.*(\n|\r\n)", ""); + } + + return expressions; + } +} diff --git a/src/main/java/org/finra/hiveqlunit/script/SingleExpressionScript.java b/src/main/java/org/finra/hiveqlunit/script/SingleExpressionScript.java new file mode 100644 index 0000000..7e66abc --- /dev/null +++ b/src/main/java/org/finra/hiveqlunit/script/SingleExpressionScript.java @@ -0,0 +1,61 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.script; + +import org.apache.spark.sql.Row; +import org.apache.spark.sql.hive.HiveContext; +import org.finra.hiveqlunit.resources.TextResource; + +/** + * Runs a single hql expression, with no heed for comments or scripts with multiple expressions in them + */ +public class SingleExpressionScript implements HqlScript { + + private String expression; + + /** + * Constructs a SingleExpressionScript + * + * The provided hql script will be run as a single expression with no pre-processing + * + * @param expressionResource a TextResource, properly containing one hql expression with no comments or such + */ + public SingleExpressionScript(TextResource expressionResource) { + expression = expressionResource.resourceText(); + } + + /** + * Runs the hql contained in the constructor given TextResource, treating it as a single expression with no comments + * + * @param hqlContext an HqlContext, as provided by spark through the TestHiveServer TestRule, used to run hql expressions + */ + @Override + public void runScript(HiveContext hqlContext) { + hqlContext.sql(expression); + } + + /** + * Runs the hql contained in the constructor given TextResource, treating it as a single expression with no comments + * + * @param hqlContext an HqlContext, as provided by spark through the TestHiveServer TestRule, used to run hql expressions + * @return a result set of Rows produced by running the hql script or expressions represented by this HqlScript + */ + @Override + public Row[] runScriptReturnResults(HiveContext hqlContext) { + return hqlContext.sql(expression).collect(); + } +} diff --git a/src/test/java/org/finra/hiveqlunit/resources/InputStreamResourceTest.java b/src/test/java/org/finra/hiveqlunit/resources/InputStreamResourceTest.java new file mode 100644 index 0000000..e2ba5c6 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/resources/InputStreamResourceTest.java @@ -0,0 +1,69 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.io.InputStream; +import java.io.PipedInputStream; +import java.io.PipedOutputStream; +import java.io.PrintStream; + +public class InputStreamResourceTest { + + @Test + public void readsResourceFromInputStreamTest() { + InputStreamResource resource = new InputStreamResource() { + @Override + public InputStream resourceStream() throws IOException { + PipedOutputStream outputStream = new PipedOutputStream(); + PipedInputStream inputStream = new PipedInputStream(outputStream); + PrintStream printer = new PrintStream(outputStream); + printer.print("Lorem ipsum doler sit amet"); + printer.close(); + + return inputStream; + } + }; + + Assert.assertEquals("Lorem ipsum doler sit amet", resource.resourceText()); + } + + @Test + public void throwsRunTimeErrorOnExceptionTest() { + InputStreamResource resource = new InputStreamResource() { + @Override + public InputStream resourceStream() throws IOException { + return new InputStream() { + @Override + public int read() throws IOException { + throw new IOException("The stream is inherently broken"); + } + }; + } + }; + + try { + resource.resourceText(); + Assert.fail(); + } catch (Exception e) { + Assert.assertEquals("Failure to load resource", e.getMessage()); + } + } +} diff --git a/src/test/java/org/finra/hiveqlunit/resources/LocalFileResourceTest.java b/src/test/java/org/finra/hiveqlunit/resources/LocalFileResourceTest.java new file mode 100644 index 0000000..d7378e6 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/resources/LocalFileResourceTest.java @@ -0,0 +1,60 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +import org.junit.Assert; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.io.PrintStream; + +public class LocalFileResourceTest { + + @Rule + public static TemporaryFolder testFolder = new TemporaryFolder(); + + @Test + public void providesCorrectInputStreamTest() { + try { + File testData = testFolder.newFile("testData.txt"); + OutputStream fileOutput = new FileOutputStream(testData); + PrintStream printStream = new PrintStream(fileOutput); + printStream.print("ABC"); + printStream.close(); + + InputStreamResource fileResource = new LocalFileResource(testData.getAbsolutePath()); + InputStream resourceStream = fileResource.resourceStream(); + + int firstByte = resourceStream.read(); + org.junit.Assert.assertEquals((int) 'A', firstByte); + + int secondByte = resourceStream.read(); + org.junit.Assert.assertEquals((int) 'B', secondByte); + + int thirdByte = resourceStream.read(); + org.junit.Assert.assertEquals((int) 'C', thirdByte); + } catch (IOException e) { + Assert.fail(); + } + } +} diff --git a/src/test/java/org/finra/hiveqlunit/resources/ResourceFolderResourceTest.java b/src/test/java/org/finra/hiveqlunit/resources/ResourceFolderResourceTest.java new file mode 100644 index 0000000..8984d85 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/resources/ResourceFolderResourceTest.java @@ -0,0 +1,46 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +import org.junit.Assert; +import org.junit.Test; + +import java.io.IOException; +import java.io.InputStream; + +public class ResourceFolderResourceTest { + + @Test + public void providesCorrectInputStreamTest() { + InputStreamResource resource = new ResourceFolderResource("/testResource.txt"); + try { + InputStream resourceStream = resource.resourceStream(); + + int firstByte = resourceStream.read(); + Assert.assertEquals((int) 'A', firstByte); + + int secondByte = resourceStream.read(); + Assert.assertEquals((int) 'B', secondByte); + + int thirdByte = resourceStream.read(); + Assert.assertEquals((int) 'C', thirdByte); + } catch (IOException e) { + Assert.fail(); + } + } + +} diff --git a/src/test/java/org/finra/hiveqlunit/resources/TextLiteralResourceTest.java b/src/test/java/org/finra/hiveqlunit/resources/TextLiteralResourceTest.java new file mode 100644 index 0000000..74c468b --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/resources/TextLiteralResourceTest.java @@ -0,0 +1,29 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources; + +import junit.framework.Assert; +import org.junit.Test; + +public class TextLiteralResourceTest { + + @Test + public void constructorSetsResourceTextTest() { + TextLiteralResource textLiteralResource = new TextLiteralResource("Foo Bar Resource"); + Assert.assertEquals("Foo Bar Resource", textLiteralResource.resourceText()); + } +} diff --git a/src/test/java/org/finra/hiveqlunit/resources/manipulation/Dos2UnixResourceTest.java b/src/test/java/org/finra/hiveqlunit/resources/manipulation/Dos2UnixResourceTest.java new file mode 100644 index 0000000..ee6f392 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/resources/manipulation/Dos2UnixResourceTest.java @@ -0,0 +1,34 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources.manipulation; + +import junit.framework.Assert; +import org.finra.hiveqlunit.resources.TextLiteralResource; +import org.finra.hiveqlunit.resources.TextResource; +import org.junit.Test; + +public class Dos2UnixResourceTest { + + @Test + public void lineEndingCorrectionTest() { + TextResource resource = new Dos2UnixResource( + new TextLiteralResource("Lorem ipsum doler sit amet\r\nLorem \ripsum doler sit\r\n") + ); + + Assert.assertEquals("Lorem ipsum doler sit amet\nLorem \ripsum doler sit\n", resource.resourceText()); + } +} diff --git a/src/test/java/org/finra/hiveqlunit/resources/manipulation/SubstituteVariableResourceTest.java b/src/test/java/org/finra/hiveqlunit/resources/manipulation/SubstituteVariableResourceTest.java new file mode 100644 index 0000000..4613b89 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/resources/manipulation/SubstituteVariableResourceTest.java @@ -0,0 +1,36 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.resources.manipulation; + +import junit.framework.Assert; +import org.finra.hiveqlunit.resources.TextLiteralResource; +import org.finra.hiveqlunit.resources.TextResource; +import org.junit.Test; + +public class SubstituteVariableResourceTest { + + @Test + public void variableSubstitutionTest() { + TextResource resource = new SubstituteVariableResource( + "variable1", + "value", + new TextLiteralResource("${variable1}Foo${variable2}Bar${variable1}") + ); + + Assert.assertEquals("valueFoo${variable2}Barvalue", resource.resourceText()); + } +} diff --git a/src/test/java/org/finra/hiveqlunit/rules/SetUpHqlTest.java b/src/test/java/org/finra/hiveqlunit/rules/SetUpHqlTest.java new file mode 100644 index 0000000..27bfbf5 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/rules/SetUpHqlTest.java @@ -0,0 +1,78 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +import junit.framework.Assert; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.hive.HiveContext; +import org.finra.hiveqlunit.script.HqlScript; +import org.junit.Test; +import org.junit.runners.model.Statement; + +import java.util.concurrent.atomic.AtomicBoolean; + +public class SetUpHqlTest { + + @Test + public void setUpHqlRunBeforeStatementTest() { + final AtomicBoolean scriptRun = new AtomicBoolean(false); + final AtomicBoolean statementCalled = new AtomicBoolean(false); + + Statement dummyStatement = new Statement() { + @Override + public void evaluate() throws Throwable { + Assert.assertTrue(scriptRun.get()); + Assert.assertFalse(statementCalled.get()); + + statementCalled.set(true); + } + }; + + TestHiveServer dummyServer = new TestHiveServer() { + @Override + public HiveContext getHiveContext() { + return null; + } + }; + + HqlScript dummyScript = new HqlScript() { + @Override + public void runScript(HiveContext hqlContext) { + Assert.assertFalse(scriptRun.get()); + Assert.assertFalse(statementCalled.get()); + + scriptRun.set(true); + } + + @Override + public Row[] runScriptReturnResults(HiveContext hqlContext) { + Assert.fail(); + return null; + } + }; + + SetUpHql setUpHql = new SetUpHql(dummyServer, dummyScript); + try { + setUpHql.apply(dummyStatement, null).evaluate(); + } catch (Throwable throwable) { + Assert.fail(); + } + + Assert.assertTrue(scriptRun.get()); + Assert.assertTrue(statementCalled.get()); + } +} diff --git a/src/test/java/org/finra/hiveqlunit/rules/TearDownHqlTest.java b/src/test/java/org/finra/hiveqlunit/rules/TearDownHqlTest.java new file mode 100644 index 0000000..edc0d2c --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/rules/TearDownHqlTest.java @@ -0,0 +1,78 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +import junit.framework.Assert; +import org.apache.spark.sql.Row; +import org.apache.spark.sql.hive.HiveContext; +import org.finra.hiveqlunit.script.HqlScript; +import org.junit.Test; +import org.junit.runners.model.Statement; + +import java.util.concurrent.atomic.AtomicBoolean; + +public class TearDownHqlTest { + + @Test + public void tearDownHqlRunAfterStatement() { + final AtomicBoolean scriptRun = new AtomicBoolean(false); + final AtomicBoolean statementCalled = new AtomicBoolean(false); + + Statement dummyStatement = new Statement() { + @Override + public void evaluate() throws Throwable { + Assert.assertFalse(scriptRun.get()); + Assert.assertFalse(statementCalled.get()); + + statementCalled.set(true); + } + }; + + TestHiveServer dummyServer = new TestHiveServer() { + @Override + public HiveContext getHiveContext() { + return null; + } + }; + + HqlScript dummyScript = new HqlScript() { + @Override + public void runScript(HiveContext hqlContext) { + Assert.assertFalse(scriptRun.get()); + Assert.assertTrue(statementCalled.get()); + + scriptRun.set(true); + } + + @Override + public Row[] runScriptReturnResults(HiveContext hqlContext) { + Assert.fail(); + return null; + } + }; + + TearDownHql tearDownHql = new TearDownHql(dummyServer, dummyScript); + try { + tearDownHql.apply(dummyStatement, null).evaluate(); + } catch (Throwable throwable) { + Assert.fail(); + } + + Assert.assertTrue(scriptRun.get()); + Assert.assertTrue(statementCalled.get()); + } +} diff --git a/src/test/java/org/finra/hiveqlunit/rules/TestDataLoaderTest.java b/src/test/java/org/finra/hiveqlunit/rules/TestDataLoaderTest.java new file mode 100644 index 0000000..9ddb040 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/rules/TestDataLoaderTest.java @@ -0,0 +1,20 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +public class TestDataLoaderTest { +} diff --git a/src/test/java/org/finra/hiveqlunit/rules/TestHiveServerTest.java b/src/test/java/org/finra/hiveqlunit/rules/TestHiveServerTest.java new file mode 100644 index 0000000..cd79828 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/rules/TestHiveServerTest.java @@ -0,0 +1,20 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.rules; + +public class TestHiveServerTest { +} diff --git a/src/test/java/org/finra/hiveqlunit/script/MultiExpressionScriptTest.java b/src/test/java/org/finra/hiveqlunit/script/MultiExpressionScriptTest.java new file mode 100644 index 0000000..4aea5aa --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/script/MultiExpressionScriptTest.java @@ -0,0 +1,20 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.script; + +public class MultiExpressionScriptTest { +} diff --git a/src/test/java/org/finra/hiveqlunit/script/ScriptSplitterTest.java b/src/test/java/org/finra/hiveqlunit/script/ScriptSplitterTest.java new file mode 100644 index 0000000..cc193d5 --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/script/ScriptSplitterTest.java @@ -0,0 +1,104 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.script; + +import junit.framework.Assert; +import org.junit.Test; + +public class ScriptSplitterTest { + + @Test + public void unixStyleLineEndings() { + String testScript = "SELECT foo FROM bar;\n" + + "SELECT lorem FROM ipsum;\n"; + + String[] statements = ScriptSplitter.splitScriptIntoExpressions(testScript); + + Assert.assertEquals(2, statements.length); + Assert.assertEquals("SELECT foo FROM bar", statements[0]); + Assert.assertEquals("SELECT lorem FROM ipsum", statements[1]); + } + + @Test + public void unixEndingInsideExpressionIgnored() { + String testScript = "SELECT foo\n FROM bar;\n"; + + String[] statements = ScriptSplitter.splitScriptIntoExpressions(testScript); + + Assert.assertEquals(1, statements.length); + Assert.assertEquals("SELECT foo\n FROM bar", statements[0]); + } + + @Test + public void windowsStyleLineEndings() { + String testScript = "SELECT foo FROM bar;\r\n" + + "SELECT lorem FROM ipsum;\r\n"; + + String[] statements = ScriptSplitter.splitScriptIntoExpressions(testScript); + + Assert.assertEquals(2, statements.length); + Assert.assertEquals("SELECT foo FROM bar", statements[0]); + Assert.assertEquals("SELECT lorem FROM ipsum", statements[1]); + } + + @Test + public void windowsEndingInsideExpressionIgnored() { + String testScript = "SELECT foo\r\n FROM bar;\n"; + + String[] statements = ScriptSplitter.splitScriptIntoExpressions(testScript); + + Assert.assertEquals(1, statements.length); + Assert.assertEquals("SELECT foo\r\n FROM bar", statements[0]); + } + + @Test + public void terminalLineEndingOptional() { + String testScript = "SELECT foo FROM bar;\n" + + "SELECT lorem FROM ipsum;"; + + String[] statements = ScriptSplitter.splitScriptIntoExpressions(testScript); + + Assert.assertEquals(2, statements.length); + Assert.assertEquals("SELECT foo FROM bar", statements[0]); + Assert.assertEquals("SELECT lorem FROM ipsum", statements[1]); + } + + @Test + public void trailingSemiColonNotPartOfScript() { + String testScript = "SELECT foo FROM bar;\n"; + + String[] statements = ScriptSplitter.splitScriptIntoExpressions(testScript); + + Assert.assertEquals(1, statements.length); + Assert.assertEquals("SELECT foo FROM bar", statements[0]); + } + + @Test + public void excludesCommentedLines() { + String testScript = "SELECT foo FROM bar;\n" + + "-- COMMENT COMMENT COMMENT\n" + + "-- COMMENT2 COMMENT2 COMMENT2\r\n" + + "SELECT lorem FROM ipsum;\n"; + + String[] statements = ScriptSplitter.splitScriptIntoExpressions(testScript); + + Assert.assertEquals(2, statements.length); + Assert.assertEquals("SELECT foo FROM bar", statements[0]); + Assert.assertEquals("SELECT lorem FROM ipsum", statements[1]); + } + +} diff --git a/src/test/java/org/finra/hiveqlunit/script/SingleExpressionScriptTest.java b/src/test/java/org/finra/hiveqlunit/script/SingleExpressionScriptTest.java new file mode 100644 index 0000000..10045be --- /dev/null +++ b/src/test/java/org/finra/hiveqlunit/script/SingleExpressionScriptTest.java @@ -0,0 +1,20 @@ +/* + * Copyright 2016 HiveQLUnit Contributors + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.finra.hiveqlunit.script; + +public class SingleExpressionScriptTest { +} diff --git a/src/test/resources/testResource.txt b/src/test/resources/testResource.txt new file mode 100644 index 0000000..48b83b8 --- /dev/null +++ b/src/test/resources/testResource.txt @@ -0,0 +1 @@ +ABC \ No newline at end of file diff --git a/userguides/MakingAnHiveQLUnitProject.md b/userguides/MakingAnHiveQLUnitProject.md new file mode 100644 index 0000000..6fc5ca9 --- /dev/null +++ b/userguides/MakingAnHiveQLUnitProject.md @@ -0,0 +1,61 @@ +# Making an HiveQLUnit Project # + +This user guide covers the steps required to start a new test suite with HiveQLUnit. + +## Required Tools ## + +HiveQLUnit is developed with git as source control and is intended for use as a maven dependency in a Java maven project. Users who do not have these tools or are not familiar with them will have a harder time using HiveQLUnit in their projects. Before proceeding further though the user guides, a user should make sure they have these tools and are comfortable with them, + +1. Java, at least version 7 +2. Git +3. Maven + +## Installing HiveQLUnit ## + +Releases of HiveQLUnit can be found in Maven Central, and Maven should automatically handle acquiring HiveQLUnit releases to use as a dependency. + +## Structuring an HiveQLUnit Project ## + +HiveQLUnit is intended for use as a Maven dependency in a Java maven project. A test suite built with HiveQLUnit needs to be structured like any other Maven project. The test suite needs a 'java' and a 'resources' folder, + + ./src/test/java + ./src/test/resources + +Classes with testing code go in src/test/java, and resources (data, scripts, ect) needed during testing go in src/test/resources. + +Configuration settings go in a pom.xml file in the root folder of the project. + + ./pom.xml + +## Configuring the Pom File ## + +The pom.xml file of the new test suite must be configured to work with HiveQLUnit. HiveQLUnit must be added as a dependency. The dependency info for the latest 1.0 release is + + + + ... + + + org.finra.hiveqlunit + hiveQLUnit + 1.0 + + + +Unit tests run with HiveQLUnit need at least 128 MB of PermGen space. When running the test suite from the command line (say with mvn clean test), the following pom.xml configuration will configure the surefire plugin to use the correct PermGen space size for unit tests + + + + + ... + + + org.apache.maven.plugins + maven-surefire-plugin + 2.18.1 + + -XX:MaxPermSize=128m + + + + \ No newline at end of file diff --git a/userguides/RunningHiveQLUnitTests.md b/userguides/RunningHiveQLUnitTests.md new file mode 100644 index 0000000..98b586f --- /dev/null +++ b/userguides/RunningHiveQLUnitTests.md @@ -0,0 +1,73 @@ +# Running HiveQLUnit Tests # + +This user guide explains how to run unit test suites constructed with HiveQLUnit. The environment does matter; different steps are needed based on operating system for example. + +## Needed JVM Arguments ## + +Several JVM arguments are needed to make HiveQLUnit test suites correctly work. This section lists the needed arguments, and the below sections 'Running from Command Line' and 'Running in an IDE' will explain how to correctly pass in the needed JVM arguments. + +### Windows ### + +Running Hive on Windows requires a number of dynamic link libraries for Hadoop. These dlls need to be present for HiveQLUnit to work properly. A full set of these dlls can be downloaded precompiled from https://codeload.github.com/srccodes/hadoop-common-2.2.0-bin/zip/master. The unziped folder can be placed anywhere, the path to it needed as a JVM argument, + + -Dhadoop.home.dir="FULL FILE PATH TO HADOOP DLLS FOLDER" + -Dhadoop.home.dir="C:/Users/k00001/hadoop/hadoop-common-2.2.0-bin-master" + +Even though the full file path is on a Windows style file system, the backslashes need to be substituted with forward slashes, or Hive will not read the file path correctly. + + C:\Users\k00001\hadoop\hadoop-common-2.2.0-bin-master -> C:/Users/k00001/hadoop/hadoop-common-2.2.0-bin-master + +### Mac/Linux ### + +Running Hive on *nix requires an extant (733 permissions) folder to use as the Hive metastore. + + -Dhive.metastore.warehouse.dir=/tmp/foo + +### PermGen Space ### + +Regardless of operating system, unit tests run with HiveQLUnit need at least 128 MB of PermGen space, + + -XX:MaxPermSize=128m + +## Running from Command Line ## + +Test suites built with HiveQLUnit can be run from the command line with + + mvn clean test + +To run from the command line (or as part of a Maven build) JVM command line arguments need to be specified in the pom.xml file to the surefire plugin: + + + + + ... + + + org.apache.maven.plugins + maven-surefire-plugin + 2.18.1 + + ARGUMENTS GO HERE + + + + + + -XX:MaxPermSize=128m -Dhive.metastore.warehouse.dir=/tmp/foo + +## Running in an IDE ## + +When unit tests are run from inside an IDE, the IDE will ignore any JVM arguments specified in the pom.xml file to the surefire plugin. The JVM arguments need to be configured into the IDE itself. + +### Intellij IDEA ### + +1. Go to 'Run' in the top bar menu +2. Under 'Run' select 'Edit Configurations...' - a menu will pop up +3. Select 'Junit' under the Defaults heading in the left side bar of the pop up menu +4. Add the needed JVM arguments to the 'VM Options' text bar +5. Click the 'Apply' button +6. Delete any configurations for individual unit tests, forcing them to all use the defaults + +### Eclipse ### + +TO DO, I do not use Eclipse, and do not actually know how to pass the args. \ No newline at end of file diff --git a/userguides/WritingHiveQLUnitTests.md b/userguides/WritingHiveQLUnitTests.md new file mode 100644 index 0000000..c464cb5 --- /dev/null +++ b/userguides/WritingHiveQLUnitTests.md @@ -0,0 +1,93 @@ +# Writing HiveQLUnit Tests # + +This user guide explains how to write a unit test using the tools provided by HiveQLUnit. HiveQLUnit unit tests are built just like any other JUnit unit tests, with a dedicated testing class containing test methods annotated with @Test. HiveQLUnit provides a number of TestRules which integrate with the JUnit framework along with other utilities for writing JUnit tests focused around Hive. + +## Resources ## + +HiveQLUnit provides utilities to access scripts and data needed during testing. + +### TextResource ### + +The TextResource interface abstracts access to resources like scripts and testing data needed in a unit test. Resources can be pulled from the src/test/resources folder bundled into the jar, from the local file system, or passed as raw text programatically. + + public interface TextResource { + + public String resourceText(); + } + + new TextLiteralResource("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") + new LocalFileResource("C:/cygwin64/home/K00001/testdata.txt") + new ResourceFolderResource("/testdata.txt") + +Many other tools provided by HiveQLUnit take a TextResource as an input. Because of this, theses tools can work with many different data sources. + +### HqlScript ### + +The HqlScript interface abstracts how to parse and run an hql script on a hive server. All implimentations of HqlScript take in a TextResource object containing the script to execute. + + new MultiExpressionScript( + new TextLiteralResource("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") + ) + + new SingleExpressionScript( + new TextLiteralResource("DROP TABLE IF EXISTS src") + ) + +## JUnit Integration ## + + @ClassRule + public static TestHiveServer hiveServer = new TestHiveServer(); + + @Rule + public static TestDataLoader loader = new TestDataLoader(hiveServer); + + @Rule + public static SetUpHql prepSrc = + new SetUpHql( + hiveServer, + new MultiExpressionScript( + new TextLiteralResource("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)") + ) + ); + + @Rule + public static TearDownHql cleanSrc = + new TearDownHql( + hiveServer, + new SingleExpressionScript( + new TextLiteralResource("DROP TABLE IF EXISTS src") + ) + ); + +TestRules are java objects which integrate with the JUnit framework and alter how unit tests are run. HiveQLUnit provides a number of TestRules which support unit tests focused around Hive. + +### TestHiveServer ### + + @ClassRule + public static TestHiveServer hiveServer = new TestHiveServer(); + +The TestHiveServer constructs an instance of HiveContext, which provides access to a disposable Hive cluster created just for testing by means of Spark. TestHiveServer needs to be the first TestRule used in a given test class, and must be annotated with the @ClassRule annotation. + +### TestDataLoader ### + + @Rule + public static TestDataLoader loader = new TestDataLoader(hiveServer); + +The TestDataLoader provides utility methods accesible from within test methods for loading data out of a TextResource and into a Hive table. + + @Test + public void test() { + loader.loadDataIntoTable("src", new LocalFileResource("C:/cygwin64/home/K00001/testdata.txt"), ""); + + HiveContext sqlContext = hiveServer.getHiveContext(); + Row[] results = sqlContext.sql("SELECT key FROM src WHERE key = 5").collect(); + Assert.assertEquals(3, results.length); + } + +### SetUpHql ### + +A test class can have multiple SetUpHql test rules. Each one takes in an HqlScript with a TextResource and runs the given script on the Hive server before every test method. All SetUpHqls are run before a @Test method, but the execution order of the different SetUpHqls is non deterministic. Be sure each SetUpHql can run independently of the others. + +### TearDownHql ### + +TearDownHql is exactly like SetUpHql, except it runs after a @Test method executes. Though the Hive server created by HiveQLUnit is disposable, things like the metastore often persist and need to be cleaned between tests. Be sure to drop tables and such after after every test to keep Hive clean. \ No newline at end of file