Skip to content

CAMEL-18792: camel-hdfs - Make integration tests runnable without a real cluster#8828

Merged
orpiske merged 3 commits intoapache:mainfrom
sekikn:CAMEL-18792
Dec 6, 2022
Merged

CAMEL-18792: camel-hdfs - Make integration tests runnable without a real cluster#8828
orpiske merged 3 commits intoapache:mainfrom
sekikn:CAMEL-18792

Conversation

@sekikn
Copy link
Contributor

@sekikn sekikn commented Dec 3, 2022

  • Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
  • Each commit in the pull request should have a meaningful subject line and body.
  • If you're unsure, you can format the pull request title like [CAMEL-XXX] Fixes bug in camel-file component, where you replace CAMEL-XXX with the appropriate JIRA issue.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Run mvn clean install -Psourcecheck in your module with source check enabled to make sure basic checks pass and there are no checkstyle violations. A more thorough check will be performed on your pull request automatically.
    Below are the contribution guidelines:
    https://github.com/apache/camel/blob/main/CONTRIBUTING.md

@github-actions
Copy link
Contributor

github-actions bot commented Dec 3, 2022

🌟 Thank you for your contribution to the Apache Camel project! 🌟

⚠️ Please note that the changes on this PR may be tested automatically.

If necessary Apache Camel Committers may access logs and test results in the job summaries!

@davsclaus
Copy link
Contributor

Thanks for helping with this. I wonder if the test used a hardcoded port number (9000 ?) then can we try using AvailablePortFinder which can find a free port number, as CI server may have "port number in use" problem.

@sekikn
Copy link
Contributor Author

sekikn commented Dec 4, 2022

Thanks for the comment! I'll fix the PR to leverage AvailablePortFinder.
In addition, I've just noticed that there's already camel-test-infra-hdfs in test-infra, so it looks better to fix it to use minicluster (it requires an external cluster now) just like hbase, rather than include minicluster directly into the integration tests. I'll update the PR.

@orpiske
Copy link
Contributor

orpiske commented Dec 4, 2022

Thanks for the comment! I'll fix the PR to leverage AvailablePortFinder. In addition, I've just noticed that there's already camel-test-infra-hdfs in test-infra, so it looks better to fix it to use minicluster (it requires an external cluster now) just like hbase, rather than include minicluster directly into the integration tests. I'll update the PR.

Thanks for your contribution!

+1 for adding it to the test-infra component. This could - potentially - also benefit the Camel Kafka Connector project which uses it.

* Replace the current Docker-based test-infra for HDFS with
  another implementation which leverages MiniDFSCluster, since
  the former seems to be unable to run all integration tests
  once with a single `mvn verify` command for some reason.

* Use AvailablePortFinder to determine Namenode's port
  for avoiding port conflict on the CI environment.
@sekikn
Copy link
Contributor Author

sekikn commented Dec 5, 2022

Thanks! Just updated the PR.

With this PR, I ensured all unit and integration tests succeeded, as follows:

$ cd test-infra/camel-test-infra-hdfs
$ mvn clean install

...

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  3.056 s
[INFO] Finished at: 2022-12-05T22:57:03+09:00
[INFO] ------------------------------------------------------------------------
$ cd -
$ cd components/camel-hdfs
$ LD_LIBRARY_PATH=/home/sekikn/.sdkman/candidates/hadoop/3.2.2/lib/native mvn clean verify

...

[INFO] --- maven-surefire-plugin:3.0.0-M5:test (default-test) @ camel-hdfs ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.camel.component.hdfs.FromFileToHdfsTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.114 s - in org.apache.camel.component.hdfs.FromFileToHdfsTest
[INFO] Running org.apache.camel.component.hdfs.HaConfigurationBuilderTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 s - in org.apache.camel.component.hdfs.HaConfigurationBuilderTest
[INFO] Running org.apache.camel.component.hdfs.HdfsConsumerTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.97 s - in org.apache.camel.component.hdfs.HdfsConsumerTest
[INFO] Running org.apache.camel.component.hdfs.HdfsInputStreamTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.033 s - in org.apache.camel.component.hdfs.HdfsInputStreamTest
[INFO] Running org.apache.camel.component.hdfs.HdfsOutputStreamTest
[INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 s - in org.apache.camel.component.hdfs.HdfsOutputStreamTest
[INFO] Running org.apache.camel.component.hdfs.HdfsProducerConsumerTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.34 s - in org.apache.camel.component.hdfs.HdfsProducerConsumerTest
[INFO] Running org.apache.camel.component.hdfs.HdfsProducerSplitTest
[INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.516 s - in org.apache.camel.component.hdfs.HdfsProducerSplitTest
[INFO] Running org.apache.camel.component.hdfs.HdfsProducerTest
[INFO] Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.486 s - in org.apache.camel.component.hdfs.HdfsProducerTest
[INFO] Running org.apache.camel.component.hdfs.kerberos.KerberosAuthenticationTest
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.008 s - in org.apache.camel.component.hdfs.kerberos.KerberosAuthenticationTest
[INFO] Running org.apache.camel.component.hdfs.kerberos.KerberosConfigurationBuilderTest
[INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 s - in org.apache.camel.component.hdfs.kerberos.KerberosConfigurationBuilderTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 46, Failures: 0, Errors: 0, Skipped: 0

...

[INFO] --- maven-failsafe-plugin:3.0.0-M4:integration-test (integration-test) @ camel-hdfs ---
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.apache.camel.component.hdfs.integration.HdfsProducerConsumerIntegrationIT

...

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.908 s - in org.apache.camel.component.hdfs.integration.HdfsProducerConsumerIntegrationIT
[INFO] Running org.apache.camel.component.hdfs.integration.HdfsConsumerIntegrationIT

...

[INFO] Tests run: 15, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 9.277 s - in org.apache.camel.component.hdfs.integration.HdfsConsumerIntegrationIT
[INFO] Running org.apache.camel.component.hdfs.integration.HdfsAppendIT

...

[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.52 s - in org.apache.camel.component.hdfs.integration.HdfsAppendIT
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 19, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] 
[INFO] --- maven-failsafe-plugin:3.0.0-M4:verify (integration-test) @ camel-hdfs ---
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:04 min
[INFO] Finished at: 2022-12-05T22:58:41+09:00
[INFO] ------------------------------------------------------------------------

@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2022

Components tested:

Total Tested Failed ❌ Passed ✅
1 1 0 1

@orpiske orpiske merged commit a0ca488 into apache:main Dec 6, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants