Skip to content

fix: using test data sample for catalog example#9372

Merged
alamb merged 4 commits intoapache:mainfrom
korowa:example_catalog
Feb 28, 2024
Merged

fix: using test data sample for catalog example#9372
alamb merged 4 commits intoapache:mainfrom
korowa:example_catalog

Conversation

@korowa
Copy link
Copy Markdown
Contributor

@korowa korowa commented Feb 27, 2024

Which issue does this PR close?

Closes #8041.

Rationale for this change

The purpose of catalog.rs to provide an example of SchemaProvider implementation, so there is no need to use submodule which may contain non-standard / non-readable .parquet files, and .csv with some sample should be enough.

What changes are included in this PR?

Replacing [parquet-]testing submodules usage in catalog example with directories and .csv containing basic sample.

Are these changes tested?

catalog.rs execution now included (back) in the CI script for examples.

Are there any user-facing changes?

No

Copy link
Copy Markdown
Contributor

@alamb alamb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @korowa -- this looks like a great improvement

I also tested this PR manually:

andrewlamb@Andrews-MacBook-Pro:~/Software/arrow-datafusion$ cargo run  --example catalog
    Finished dev [unoptimized + debuginfo] target(s) in 0.17s
     Running `target/debug/examples/catalog`
[2024-02-28T00:58:41Z INFO  catalog] querying table 2.csv from schema_a
[2024-02-28T00:58:41Z INFO  catalog] query completed
+-----+-------+
| key | value |
+-----+-------+
| 1   | foo   |
| 2   | bar   |
| 3   | baz   |
+-----+-------+
[2024-02-28T00:58:41Z INFO  catalog] querying table 3.csv from schema_a
[2024-02-28T00:58:41Z INFO  catalog] query completed
+-----+-------+
| key | value |
+-----+-------+
| 1   | foo   |
| 2   | bar   |
| 3   | baz   |
+-----+-------+
[2024-02-28T00:58:41Z INFO  catalog] querying table 4.csv from schema_a
[2024-02-28T00:58:41Z INFO  catalog] query completed
+-----+-------+
| key | value |
+-----+-------+
| 1   | foo   |
| 2   | bar   |
| 3   | baz   |
+-----+-------+
[2024-02-28T00:58:41Z INFO  catalog] querying table 1.csv from schema_a
[2024-02-28T00:58:41Z INFO  catalog] query completed
+-----+-------+
| key | value |
+-----+-------+
| 1   | foo   |
| 2   | bar   |
| 3   | baz   |
+-----+-------+
[2024-02-28T00:58:41Z INFO  catalog] querying table 0.csv from schema_a
[2024-02-28T00:58:41Z INFO  catalog] query completed
+-----+-------+
| key | value |
+-----+-------+
| 1   | foo   |
| 2   | bar   |
| 3   | baz   |
+-----+-------+
[2024-02-28T00:58:41Z INFO  catalog] dropping table 2.csv

Comment thread datafusion-examples/Cargo.toml
Comment thread datafusion-examples/examples/catalog.rs
Comment thread datafusion-examples/examples/catalog.rs
@alamb alamb merged commit 03e8323 into apache:main Feb 28, 2024
@alamb
Copy link
Copy Markdown
Contributor

alamb commented Feb 28, 2024

Thanks again @korowa

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Error from example catalog

2 participants