invalid parquet version error for parquet files generated via python script #144

[saritvakrat]

Hi, I am trying to read parquet files that are in S3 and were generated via python script.
I get the following error:
Error: thrown: "invalid parquet version"
When I am trying to read similar file but the file was generated via spark - it manages to digest the file and read it.

I am also able to parse the python file and open it in a parquet viewer

Any idea why? the file is parquet lvl 2

Full error:

(node:41711) V8: /Users/saritvakrat/Documents/automation/be_automation/node_modules/brotli/build/encode.js:34 Linking failure in asm.js: Unexpected stdlib member (Use node --trace-warnings ...` to show where the warning was created)
console.error
Error parsing Parquet file: invalid parquet version

  39 |         return records;
  40 |     } catch (error) {
> 41 |         console.error('Error parsing Parquet file:', error);
     |                 ^
  42 |         throw error; // Rethrow the error to be handled by the caller
  43 |     }
  44 | }`
  
  Packages:
      "parquetjs": "^0.11.2",
"@types/parquetjs": "^0.10.6",

My function:
``export async function parseParquetFile(filePath: string): Promise<any[]> {
try {
// create new ParquetReader
const reader = await ParquetReader.openFile(filePath) as any;
// create a new cursor
const cursor = reader.getCursor();
const records = [];
// read all records from the file and print them
let record = await cursor.next();
while (record !== null) {
records.push(record);
record = await cursor.next();
}
await reader.close();
return records;
} catch (error) {
console.error('Error parsing Parquet file:', error);
throw error; // Rethrow the error to be handled by the caller
}
}

`async parseSingleParquetFromS3(bucketName: string, key: string | null | undefined): Promise<any[]> {
if (!bucketName || !key) {
throw new Error('S3 client or bucket name is not provided');
}

    const getObjectCommand = new GetObjectCommand({
        Bucket: bucketName,
        Key: key
    });

    let objectResponse;
    try {
        objectResponse = await this.s3Client.send(getObjectCommand);
    } catch (error) {
        console.error(`Error fetching object from S3: ${error}`);
        throw error;
    }

    const objectData = objectResponse.Body;
    if (!(objectData instanceof Readable)) {
        throw new Error('Object data is not a readable stream');
    }

    const fileName = key.split('/').pop() || 'temp.parquet';
    const tempFilePath = join(tmpdir(), fileName);

    try {
        await pipeline(objectData, createWriteStream(tempFilePath));
        return await parseParquetFile(tempFilePath);
    } catch (error) {
        console.error(`Error in streaming data to file: ${error}`);
        throw error;
    }
}`

reading python generated file:

reading spark generated file: