TianYi ZHU Fantoccini

  • Commonwealth Bank
  • Sydney, Australia
  • Joined on
@Fantoccini
Parquet reading throws ParquetDecodingException .
Fantoccini commented on issue CommBank/ebenezer#93
@Fantoccini

Update: it's a data schema inconsistent issue, has informed the data team.

Fantoccini commented on issue CommBank/ebenezer#93
@Fantoccini

Update: it might be: data schema inconsistent + outdated code in jar + cluster settings will attach a remote debugger on cluster and see if I can f…

Fantoccini commented on issue CommBank/ebenezer#93
@Fantoccini

update: I copied the file from hdfs to local disk, run with --local and the same code, everything works fine.

Fantoccini commented on issue CommBank/ebenezer#93
@Fantoccini

I thought it only failed on some file, so I tried to remove those files in load code on each run. After I removed 6 'bad' files, the M/R job still …

@Fantoccini
Parquet reading throws ParquetDecodingException .
Fantoccini commented on issue CommBank/ebenezer#93
@Fantoccini

Hi, I encountered this error as well. I'm trying to read data from hive table pr_hls.loan_master_szlnmst, on hdfs, the paths are: /prod/view/wareho…