Zero copy import when the schema is known

[darabos]

Resolves #258.

No import button! The corresponding Python code is:

lk.importParquet(eager='no', filename='/home/darabos/eg.parquet', schema='name: String, age: Double')

Outstanding issues:

• Currently you can only "import" a file this way once. LynxKite assumes it will never change. This could be avoided with a version parameter, same as its done with export operations.
• Add the three parameters: imported_columns, limit, and sql.
• Tests, documentation.

[darabos]

Tuck also wants to read Date and Timestamp columns. I've added these.

This is useful outside of this PR too. But it will be easier to test them together. Hopefully we end up merging both changes.

[tuckging]

Tuck also wants to read Date and Timestamp columns. I've added these.

This is useful outside of this PR too. But it will be easier to test them together. Hopefully we end up merging both changes.

Thanks @darabos, seems to work great! 🙌 We can merge this for now, but I've come across another datatype below. If it's trivial to fix, we can include it in this PR as well!

decimal(8,0)
decimal(18,2)
decimal(12,0)
decimal(9,2)

[tuckging]

PS you might want to pin openjdk in conda-env.yml for CI to start passing again

- sbt
- openjdk==11.0.15 # if left unpinned, sbt now brings in unsupported jdk17

[darabos]

PS you might want to pin openjdk in conda-env.yml for CI to start passing again

- sbt
- openjdk==11.0.15 # if left unpinned, sbt now brings in unsupported jdk17

Haha, @lacca0 just discovered the same 2 minutes ago! We will look into changing the code to be compatible with Java 17. But this workaround is very useful until then!

Thanks @darabos, seems to work great! 🙌

Awesome, thanks a lot for testing!

We can merge this for now, but I've come across another datatype below. If it's trivial to fix, we can include it in this PR as well!

Do you have a small test file or pyspark script to generate one? I want to make sure I'm targeting the right type.

[tuckging]

Do you have a small test file or pyspark script to generate one? I want to make sure I'm targeting the right type.

from pyspark.sql import functions as F
spark.range(100).withColumn('decimal_col', F.rand().cast('decimal(18,2)'))

[darabos]

Thanks! I've changed the code to support all types that Spark supports. I let Spark parse what you enter.

If the table does not match the specified schema you get an error. I changed the error formatting a bit to make sure this includes the important bit about what didn't match.

I've also added documentation. I'll add unit tests on Monday and then we can merge it.

[darabos]

@lacca0 I'm going to merge this PR to simplify merge conflicts with the 1-jar change. But it's still open for your comments! (I'll just have to address them in a separate PR.) Thanks!