-
Notifications
You must be signed in to change notification settings - Fork 19
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Cannot read properties of undefined (reading 'fromIPCStream') #471
Comments
The API changed after 0.5.0. In 0.5 the |
Thanks for this hint (it is really good that you answer so fast to the issues 👍). Now I have another Problem with the writeParquet function. The following lines make trouble
as writeParquet is expecting a Table:
So I tried
and importing from 'parquet-wasm/node/arrow1' which compiles. However, this produces an empty schema. Therefore, the question is, how to call writeParquet from the return of tableToIPC(rainfall, 'stream')? BTW: I changed the apache-arrow version to 13.0.0 as this version is also used in parquet-wasm 0.5.0 |
The best suggestion is to use the typescript types to guide you. This is working for me with 0.5.0 import { tableFromArrays, tableToIPC } from "apache-arrow";
import * as parquet from "parquet-wasm/node/arrow1";
import { writeFileSync } from "fs";
// Create Arrow Table in JS
const LENGTH = 2000;
const rainAmounts = Float32Array.from({ length: LENGTH }, () =>
Number((Math.random() * 20).toFixed(1))
);
const rainDates = Array.from(
{ length: LENGTH },
(_, i) => new Date(Date.now() - 1000 * 60 * 60 * 24 * i)
);
const rainfall = tableFromArrays({
precipitation: rainAmounts,
date: rainDates,
});
// Write Arrow Table to Parquet
const writerProperties = new parquet.WriterPropertiesBuilder()
.setCompression(parquet.Compression.ZSTD)
.build();
const arrowWasmTable = parquet.Table.fromIPCStream(
tableToIPC(rainfall, "stream")
);
const parquetBuffer = parquet.writeParquet(arrowWasmTable, writerProperties);
writeFileSync("out.parquet", parquetBuffer); |
Hello Kyle, thanks for your support and time :). Indeed, your code works, and it turned out that my code, which I posted earlier, works as well. However, the schema is empty when I inspect the table by setting a break point after calling 'tableFromIPC(readParquet(parquetBuffer));'. |
The entire table is empty. |
Perfect, 'tableFromIPC(readParquet(parquetBuffer).intoIPCStream())' worked. |
BTW: Do you have any example to use 'readParquetStream'? If I do this with the stream:
I get the following runtime error:
|
I think the right abstract is a class like |
Setup:
Typescript == 4.9.5
node == 20.0.0
theia == 1.45.0
Webpack == 5.90.3
parquet-wasm == 0.5.0
apache-arrow == 15.0.0
I tried the example code:
and get the error message Cannot read properties of undefined (reading 'fromIPCStream'). Inspecting the ipcStream at runtime reveals that ipcStream is an Uint8Array with 24400 entries.
My imports:
Do you have any idea of what is going wrong?
The text was updated successfully, but these errors were encountered: