CsvIO limitation of HeaderDecoder and RowDecoder

[albertols]

Hi All,

Following:

case class User(name: String, age: Int) implicit val decoder = HeaderDecoder.decoder("fullName", "userAge")(User.apply _) val users: SCollection[User] = scioContext.csvFile(path)
from https://github.com/spotify/scio/blob/main/scio-extra/src/main/scala/com/spotify/scio/extra/csv/CsvIO.scala#L58

I am trying to use Case Classes with more than 22, fields but I am seeing this limitation

From kantan.csv library, I do not where exactly is coming from or if I am missing something here.

https://github.com/nrinaudo/kantan.csv/blob/v0.7.0/core/shared/src/main/boilerplate/kantan/csv/GeneratedHeaderDecoders.scala.template

Thanks guys!

Alberto

P.S: Same with RowDecoder:
https://github.com/nrinaudo/kantan.csv/blob/v0.7.0/core/shared/src/main/boilerplate/kantan/csv/GeneratedRowDecoders.scala.template

[RustedBones]

Hi @albertols

You are facing a very well known scala issue with the 22 limit.

Even if the 22 limit has been lifted on case class for a while, most libraries (including scio) only define their API to a maximum 22 parameters.

I fear you'll have to use a custom decoder implementation in this case.

Closing the issue since this is not scio specific.

[albertols]

Thanks for the fast response @RustedBones ,

Maybe the 22 limitation could be included in the comments. I am able to use case classes with more than 22 fields, e.g:

saveAsTypedBigQuery

I will try to give it a go ;) I do not know if https://github.com/tototoshi/scala-csv or similar could dodge the limitation. Maybe, as future enhancement CsvIO could deal with other Decoders/Encoders to avoid this limitation.

Cheers!