transformAlignments cannot repartition files

[akmorrow13]

~/ADAM/adam/bin/adam-submit --packages org.apache.parquet:parquet-avro:1.8.2 --master yarn-client --num-executors 8 --executor-memory 20g -- transformAlignments /data/platinum/NA12877_S1.bam /data/platinum/transformedAlignments/NA12877_S1.bam.adam -repartition 200

or ~/ADAM/adam/bin/adam-submit --master yarn-client --num-executors 8 --executor-memory 20g -- transformAlignments -repartition 200 /data/platinum/NA12877_S1.bam /data/platinum/transformedAlignments/NA12877_S1.bam.adam

do not repartition

[fnothaft]

What happens instead? What happens if you use -coalesce instead? If you're decreasing the number of partitions, you'll generally want to use coalesce instead of repartition, as coalesce is shuffle free.

[akmorrow13]

I used repartition. The result was generating ~1200 partitions, instead of 200. It's hard to tell the starting partitions from a bam file, so I guess one would calculate the estimated partitions based on resources then call repartition or coalesce accordingly?

[fnothaft]

I used repartition. The result was generating ~1200 partitions, instead of 200.

Can you send me the application ID offline? I'd like to take a look.

It's hard to tell the starting partitions from a bam file

You should get roughly partitions = file_size / block_size. We have the cluster configured with a 128MB block size, so a 128GB file would have approximately 1024 partitions.

[akmorrow13]

Sent @fnothaft ! That would explain the large number of partitions then, given the file was ~120GB.

[fnothaft]

I think we debugged this locally. Closing. Please reopen @akmorrow13 if I was wrong.