reg_file with index via s3

[vladsavelyev]

Hi,

Trying to register a bam file via S3, but getting error, even though the corresponding .bai file is there:

docker exec -it ngbcore bash -c 'ngb reg_file hg38 s3://umccr-primary-data-dev/COLO829BL/LR_2.2.2/phased_possorted_bam.bam -n input_BAM'
05:26:14.138 [main] ERROR com.epam.ngb.cli.app.Application - Index file is required for file [s3://umccr-primary-data-dev/COLO829BL/LR_2.2.2/phased_possorted_bam.bam].

aws s3 ls s3://umccr-primary-data-dev/COLO829BL/LR_2.2.2/phased_possorted_bam.bam
2018-07-20 23:05:48 161758995212 phased_possorted_bam.bam
2018-07-20 23:05:48   10389304 phased_possorted_bam.bam.bai

Can you advice please? :)

Vlad

/cc @brainstorm

[SilinPavel]

Hello, @vladsaveliev!
Thank you for your issue, currently we support S3 only as unregistered files (by s3 signed url).
Actually we plan to develop full s3 support (via AWS SDK) due to end of the year.

[brainstorm]

Fantastic! Thanks Silin for the feedback! We had a quick look at that code, one small hint you might want to have in mind: make sure the S3 support also has IAM instance profile (instances without secret keys), since it's much more secure and automatable.

[brainstorm]

@SilinPavel More precisely, we mean this:

https://github.com/brainstorm/NGB/commit/a91798ce6d8f1afb031b350a02c8ba98ff15f477#diff-7d2cf1651d997044f9f512e0a37b5013

It basically picks up what's available where the client is running, avoiding burning secrets into AWS AMI's.