Skip to content

Commit

Permalink
[Module] Optimize the module structure of the xtable
Browse files Browse the repository at this point in the history
  • Loading branch information
daragu authored and vinishjail97 committed May 21, 2024
1 parent 3352098 commit d991e75
Show file tree
Hide file tree
Showing 196 changed files with 29 additions and 28 deletions.
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,7 @@ catalogOptions: # all other options are passed through in a map
key1: value1
key2: value2
```
5. run with `java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml [--hadoopConfig hdfs-site.xml] [--convertersConfig converters.yaml] [--icebergCatalogConfig catalog.yaml]`
5. run with `java -jar xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml [--hadoopConfig hdfs-site.xml] [--convertersConfig converters.yaml] [--icebergCatalogConfig catalog.yaml]`
The bundled jar includes hadoop dependencies for AWS, Azure, and GCP. Sample hadoop configurations for configuring the converters
can be found in the [xtable-hadoop-defaults.xml](https://github.com/apache/incubator-xtable/blob/main/utilities/src/main/resources/xtable-hadoop-defaults.xml) file.
The custom hadoop configurations can be passed in with the `--hadoopConfig [custom-hadoop-config-file]` option.
Expand Down
6 changes: 3 additions & 3 deletions demo/start_demo.sh
Original file line number Diff line number Diff line change
Expand Up @@ -18,9 +18,9 @@
## Create the require jars for the demo and copy them into a directory we'll mount in our notebook container
cd .. && mvn install -am -pl core -DskipTests -T 2
mkdir -p demo/jars
cp hudi-support/utils/target/hudi-utils-0.1.0-SNAPSHOT.jar demo/jars
cp api/target/xtable-api-0.1.0-SNAPSHOT.jar demo/jars
cp core/target/xtable-core-0.1.0-SNAPSHOT.jar demo/jars
cp xtable-hudi-support/xtable-hudi-support-utils/target/xtable-hudi-support-utils-0.1.0-SNAPSHOT.jar demo/jars
cp xtable-api/target/xtable-api-0.1.0-SNAPSHOT.jar demo/jars
cp xtable-core/target/xtable-core-0.1.0-SNAPSHOT.jar demo/jars

cd demo
docker-compose up
10 changes: 5 additions & 5 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,10 @@
<packaging>pom</packaging>

<modules>
<module>api</module>
<module>core</module>
<module>utilities</module>
<module>hudi-support</module>
<module>xtable-api</module>
<module>xtable-core</module>
<module>xtable-utilities</module>
<module>xtable-hudi-support</module>
</modules>

<properties>
Expand Down Expand Up @@ -89,7 +89,7 @@
</dependency>
<dependency>
<groupId>org.apache.xtable</groupId>
<artifactId>hudi-utils</artifactId>
<artifactId>xtable-hudi-support-utils</artifactId>
<version>${project.version}</version>
</dependency>

Expand Down
2 changes: 1 addition & 1 deletion website/docs/biglake-metastore.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,7 +117,7 @@ catalogOptions:
From your terminal under the cloned Apache XTable™ (Incubating) directory, run the sync process using the below command.

```shell md title="shell"
java -cp utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar:/path/to/downloaded/biglake-catalog-iceberg1.2.0-0.1.0-with-dependencies.jar org.apache.xtable.utilities.RunSync --datasetConfig my_config.yaml --icebergCatalogConfig catalog.yaml
java -cp xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar:/path/to/downloaded/biglake-catalog-iceberg1.2.0-0.1.0-with-dependencies.jar org.apache.xtable.utilities.RunSync --datasetConfig my_config.yaml --icebergCatalogConfig catalog.yaml
```

:::tip Note:
Expand Down
2 changes: 1 addition & 1 deletion website/docs/fabric.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,7 +98,7 @@ An example hadoop configuration for authenticating to ADLS storage account is as
```

```shell md title="shell"
java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml --hadoopConfig hadoop.xml
java -jar xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml --hadoopConfig hadoop.xml
```

Running the above command will translate the table `people` in Iceberg or Hudi format to Delta Lake format. To validate
Expand Down
2 changes: 1 addition & 1 deletion website/docs/features-and-limitations.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,7 @@ This sync provides users with the following:
- the metadata table (`hoodie.metadata.enable=true`) and
- hive style partitioning (`hoodie.datasource.write.hive_style_partitioning=true`) wherever applicable when reading the data.
- Be sure to enable `parquet.avro.write-old-list-structure=false` for proper compatibility with lists when syncing from Hudi to Iceberg.
- When using Hudi as the source for an Iceberg target, you may require field IDs set in the parquet schema. To enable that, follow the instructions [here](https://github.com/apache/incubator-xtable/tree/main/hudi-support/extensions).
- When using Hudi as the source for an Iceberg target, you may require field IDs set in the parquet schema. To enable that, follow the instructions [here](https://github.com/apache/incubator-xtable/tree/main/xtable-hudi-support/xtable-hudi-support-extensions).

### Delta
- When using Delta as the source for an Iceberg target, you may require field IDs set in the parquet schema. To enable that, follow the instructions for enabling column mapping [here](https://docs.delta.io/latest/delta-column-mapping.html).
Expand Down
2 changes: 1 addition & 1 deletion website/docs/glue-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,7 +84,7 @@ Replace with appropriate values for `sourceFormat`, `tableBasePath` and `tableNa
From your terminal under the cloned xtable directory, run the sync process using the below command.

```shell md title="shell"
java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
java -jar xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
```

:::tip Note:
Expand Down
2 changes: 1 addition & 1 deletion website/docs/hms.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,7 +88,7 @@ datasets:

From your terminal under the cloned Apache XTable™ (Incubating) directory, run the sync process using the below command.
```shell md title="shell"
java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
java -jar xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
```

:::tip Note:
Expand Down
2 changes: 1 addition & 1 deletion website/docs/how-to.md
Original file line number Diff line number Diff line change
Expand Up @@ -351,7 +351,7 @@ Authentication for GCP requires service account credentials to be exported. i.e.
In your terminal under the cloned Apache XTable™ (Incubating) directory, run the below command.

```shell md title="shell"
java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
java -jar xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
```

**Optional:**
Expand Down
2 changes: 1 addition & 1 deletion website/docs/unity-catalog.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ datasets:
From your terminal under the cloned Apache XTable™ (Incubating) directory, run the sync process using the below command.

```shell md title="shell"
java -jar utilities/target/utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
java -jar xtable-utilities/target/xtable-utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
```

:::tip Note:
Expand Down
2 changes: 1 addition & 1 deletion api/pom.xml → xtable-api/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>xtable-api</artifactId>
<name>api</name>
<name>xtable-api</name>

<parent>
<groupId>org.apache.xtable</groupId>
Expand Down
4 changes: 2 additions & 2 deletions core/pom.xml → xtable-core/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>xtable-core</artifactId>
<name>core</name>
<name>xtable-core</name>

<parent>
<groupId>org.apache.xtable</groupId>
Expand All @@ -35,7 +35,7 @@
</dependency>
<dependency>
<groupId>org.apache.xtable</groupId>
<artifactId>hudi-utils</artifactId>
<artifactId>xtable-hudi-support-utils</artifactId>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
Expand Down
File renamed without changes.
7 changes: 4 additions & 3 deletions hudi-support/pom.xml → xtable-hudi-support/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -25,11 +25,12 @@
<version>0.1.0-SNAPSHOT</version>
</parent>

<artifactId>hudi-support</artifactId>
<artifactId>xtable-hudi-support</artifactId>
<packaging>pom</packaging>


<modules>
<module>extensions</module>
<module>utils</module>
<module>xtable-hudi-support-utils</module>
<module>xtable-hudi-support-extensions</module>
</modules>
</project>
File renamed without changes.
Original file line number Diff line number Diff line change
Expand Up @@ -19,18 +19,18 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>hudi-extensions</artifactId>
<artifactId>xtable-hudi-support-extensions</artifactId>

<parent>
<groupId>org.apache.xtable</groupId>
<artifactId>hudi-support</artifactId>
<artifactId>xtable-hudi-support</artifactId>
<version>0.1.0-SNAPSHOT</version>
</parent>

<dependencies>
<dependency>
<groupId>org.apache.xtable</groupId>
<artifactId>hudi-utils</artifactId>
<artifactId>xtable-hudi-support-utils</artifactId>
</dependency>
<dependency>
<groupId>org.apache.xtable</groupId>
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,11 +19,11 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<artifactId>hudi-utils</artifactId>
<artifactId>xtable-hudi-support-utils</artifactId>

<parent>
<groupId>org.apache.xtable</groupId>
<artifactId>hudi-support</artifactId>
<artifactId>xtable-hudi-support</artifactId>
<version>0.1.0-SNAPSHOT</version>
</parent>

Expand Down
2 changes: 1 addition & 1 deletion utilities/pom.xml → xtable-utilities/pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>utilities</artifactId>
<artifactId>xtable-utilities</artifactId>

<dependencies>
<dependency>
Expand Down
File renamed without changes.

0 comments on commit d991e75

Please sign in to comment.