Skip to content

[HUDI-7822] Resolve the conflicts between mixed hdfs and local path in Flink tests#10931

Merged
yihua merged 2 commits intoapache:masterfrom
hehuiyuan:fix-unified_hadoopconf
May 31, 2024
Merged

[HUDI-7822] Resolve the conflicts between mixed hdfs and local path in Flink tests#10931
yihua merged 2 commits intoapache:masterfrom
hehuiyuan:fix-unified_hadoopconf

Conversation

@hehuiyuan
Copy link
Contributor

@hehuiyuan hehuiyuan commented Mar 28, 2024

Change Logs

Resolve the conflicts for which may be mixed hdfs and local path together

org.apache.flink.table.catalog.exceptions.CatalogException: Catalog hudi path /var/folders/k1/65gcjk_92ws2bjh3ftpz33fc0000gp/T/junit845658121822910144 does not exist.

	at org.apache.hudi.table.catalog.HoodieCatalog.open(HoodieCatalog.java:122)
	at org.apache.hudi.table.catalog.TestHoodieCatalog.beforeEach(TestHoodieCatalog.java:167)

Impact

It can use hdfs and local path to run test case.

The tempdir is creatd by local path. But hoodiecatalog may use hdfs path if the hadoop env exist

Risk level

none

Documentation Update

none

Contributor's checklist

  • Read through contributor's guide
  • Change Logs and Impact were stated clearly
  • Adequate tests were added if applicable
  • CI passed

@github-actions github-actions bot added the size:S PR with lines of changes in (10, 100] label Mar 28, 2024
@CTTY
Copy link
Contributor

CTTY commented Mar 28, 2024

This looks fine. But just being curious here, why do we need to unify how we get hadoopConf? Is there any bigger refactoring/feature on the way?

@hehuiyuan
Copy link
Contributor Author

hehuiyuan commented Mar 28, 2024

This looks fine. But just being curious here, why do we need to unify how we get hadoopConf? Is there any bigger refactoring/feature on the way?

Hi , it only use the mehtod HadoopConfigurations.getHadoopConf to create Filesystem which can keep consistency

@hehuiyuan hehuiyuan changed the title [Test] Unify to get the hadoopConf which can use hdfs and local path [Test] Resolve the conflicts for which may be mixed hdfs and local path together Mar 28, 2024
@yihua yihua changed the title [Test] Resolve the conflicts for which may be mixed hdfs and local path together [HUDI-7822] Resolve the conflicts for which may be mixed hdfs and local path together May 31, 2024
@yihua yihua force-pushed the fix-unified_hadoopconf branch from e09914c to 3d87799 Compare May 31, 2024 20:31
@yihua yihua changed the title [HUDI-7822] Resolve the conflicts for which may be mixed hdfs and local path together [HUDI-7822] Resolve the conflicts between mixed hdfs and local path in Flink tests May 31, 2024
@hudi-bot
Copy link
Collaborator

CI report:

Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

@yihua
Copy link
Contributor

yihua commented May 31, 2024

Azure CI is green.
Screenshot 2024-05-31 at 15 33 50

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:S PR with lines of changes in (10, 100]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants