Reharmonising a file with error

[ljwh2]

GCST002047 was not harmonised successfully because our harmonisation pipeline cannot recognise the column “Effect_Allele”. The harmonisation pipeline reads the “effect_allele” column in the input file to harmonise the variant. However, all data in this column is NA. This is the reason why all variants give hm 14. If we change the header of this file, it should be able to be harmonised. (same as other_allele)

Please fix the file and re-qeue for harmonisation

[karatugo]

When unzipped two files appeared. I fixed the header for both but the metadata yaml files are missing. Since the study is old, the data is not available from the ingest api.

[karatugo]

• Create metadata yaml files that contain GCST ID and genome assembly for the harmonisation.
• Submit the harmonisation

[karatugo]

Submitted to codon with the submission script /hps/software/users/parkinso/spot/gwas/prod/scripts/cron/start_harmonisation_pre_standard_goci1226.sh

Job <92779129> is submitted to default queue <standard>.

[karatugo]

• Compare two studies in FTP ("EduYears" and "College")
• If they are the same, replace the zipped file with the correct one - Moved SSGAC_College_Rietveld2013_publicrelease.txt to GCST002001-GCST003000/GCST002047 and removed the zipped file
• Harmonise with pre_gwas_ssf
• Also, upload only one harmonised file with the correct title

[karatugo]

• Use variant_id in the header (as per pre_gwas_ssf standard)
• Rename the file with their GCST_ID.txt for harmonisation

[karatugo]

Using the script at /hps/software/users/parkinso/spot/gwas/prod/scripts/cron/start_harmonisation_pre_standard_goci1226.sh

Job <93028342> is submitted to default queue <standard>.

[karatugo]

Added chromosome and bas_pair_location columns filled with NA and submitted again.

Job <93126943> is submitted to default queue <standard>.

[karatugo]

Harmonised files, metadata files, running logs and .tbi files are copied to the respective harmonised directories.

• http://ftp.ebi.ac.uk/pub/databases/gwas/summary_statistics/GCST002001-GCST003000/GCST002047/harmonised/
• http://ftp.ebi.ac.uk/pub/databases/gwas/summary_statistics/GCST008001-GCST009000/GCST008396/harmonised/

[sprintell]

This is confirmed done, @earlEBI will double check

[earlEBI]

Reopening as the yaml files do not look quite right. (is_harmonised = false).
Also, the .tbi files should be renamed .tbi.gz.

[karatugo]

Fixed the following fields:

genome_assembly: GRCh38
is_harmonised: true
is_sorted: true

@earlEBI Could you check again please? Thanks.

[ljwh2]

@earlEBI please confirm

[earlEBI]

The yamls are only five lines long. Should they not contain more detail?

[karatugo]

I thought that's because it's a very old submission. And also they are not available in the ingest api. @sajo-ebi

https://www.ebi.ac.uk/gwas/ingest/api/v2/studies/GCST008396
https://www.ebi.ac.uk/gwas/ingest/api/v2/studies/GCST002047

[sprintell]

Old studies are meant to be retrieved fromthe public rest API: https://www.ebi.ac.uk/gwas/rest/api/studies/GCST008396

[karatugo]

TODO: Update sumstats tools so that we fetch the REST API if Ingest API does not return any data.

[sprintell]

Harmonization done, but yaml file has some missing data.

[karatugo]

Regenerated YAML files for GCST002047 and GCST008396. Expect them in the public ftp in 2 days.

[karatugo]

YAML files are in staging FTP but not in public FTP. The reason why it didn't sync is in our ftp-sync code, we only filter the files that start with 'GCST*'. See https://github.com/EBISPOT/gwas-utils/blob/6fbf2c7a6d6fdfc79e0b8c2d1e74539bb1073303/ftpSummaryStatsScript/ftp_sync.py#L186-L188

Will renamed files, expect them in the public ftp in 2 days.

[ljwh2]

Agreed to keep original files as per old guidelines