RTCGA data to ExperimentHub

[MarcinKosinski]

Hi Marcin,

I'm ready to starting putting the RTCGA data sets in ExperimentHub. The idea is that we'll create a data package, similar to the usual Experimental Data packages, that will have a little more documentation and background. The data will be in an S3 bucket - it's ok if the data.frames live in S3 and we transform them to SummarizedExperiment when the user invokes '[[' on the resource, e.g.,

eh = ExperimentHub()
eh[["EH1234"]]

To get started please do the following:

1. Let me know what you plan to name your data package, e.g., RTCGAData or just RTCGA etc. Once we have this we'll make an S3 bucket of the same name and give you permissions to upload the data.

2. Put together the data package following instructions here, specifically section 2.1 'Add New Resources':

http://www.bioconductor.org/packages/3.4/bioc/vignettes/ExperimentHubData/inst/doc/ExperimentHubData.html

Here is a sample package (data are in ExperimentHub) you can use as a template:

http://www.bioconductor.org/packages/3.3/data/experiment/html/GSE62944.html

Thanks.

[MarcinKosinski]

2016-05-31 15:30 GMT+02:00 Valerie Obenchain vobencha@gmail.com:

Hi Marcin,

Hello Valerie,

It's good to hear from you as I have a scheduled Skype call for tommorow to talk about uploading new data package/packages to ExperimentHub with my collaborator, Witold Chodor, from RTCGA project https://github.com/orgs/RTCGA/people (CC'd other contributors).

I'm ready to starting putting the RTCGA data sets in ExperimentHub. The idea is that we'll create a data package, similar to the usual Experimental Data packages, that will have a little more documentation and background. The data will be in an S3 bucket - it's ok if the data.frames live in S3 and we transform them to SummarizedExperiment when the user invokes '[[' on the resource, e.g.,

eh = ExperimentHub()
eh[["EH1234"]]

To get started please do the following:

1) Let me know what you plan to name your data package, e.g., RTCGAData or just RTCGA etc. Once we have this we'll make an S3 bucket of the same name and give you permissions to upload the data.

I think that "RTCGA.data" is a good name to start. I'll create there "RTCGA.data_type.date_of_release" data.frames / SummarizedExperiments.

2) Put together the data package following instructions here, specifically section 2.1 'Add New Resources':

http://www.bioconductor.org/packages/3.4/bioc/vignettes/ExperimentHubData/inst/doc/ExperimentHubData.html

Here is a sample package (data are in ExperimentHub) you can use as a template:

http://www.bioconductor.org/packages/3.3/data/experiment/html/GSE62944.html

Are there any size limits for such packages? I would like to combine datasets from, among others, RTCGA.methylation and RTCGA.rnaseq Experiment Data packages which both together exceed 16 GBs and might not build as 1 binary file as they exceed my RAM limits.

Maybe it's a good idea to upload more than 1 package to ExperimentHub like

• RTCGA.clinical.20160128 as refreshed, newer version of RTCGA.clinical Experiment Data Package

• RTCGA.mutations.20160128 as refreshed, newer version of RTCGA.mutatios Experiment Data Package

• RTCGA.rnaseq.20160128 as refreshed, newer version of RTCGA.rnaseq Experiment Data Package

• RTCGA.methylation.20160128 as refreshed, newer version of RTCGA.methylation Experiment Data Package

• RTCGA.RPPA.20160128 as refreshed, newer version of RTCGA.RPPA Experiment Data Package

• RTCGA.miRNASeq.20160128 as refreshed, newer version of RTCGA.miRNASeq Experiment Data Package

• RTCGA.mRNA.20160128 as refreshed, newer version of RTCGA.mRNA Experiment Data Package

• RTCGA.CNV.20160128 as refreshed, newer version of RTCGA.CNV Experiment Data Package

  Thanks.
  Valerie

Would you rather continue this issue via e-mails or maybe we can open na issue on our repository at GitHub https://github.com/RTCGA/RTCGA/issues so that more people can benefit from our private correspondence :)?

Best,
Marcin

[MarcinKosinski]

Valerie:

I would try to keep the file size <= 5GB. How many data.frames do you
have for each of these categories? e.g., how many for clinical, for
mutations, etc.?

[MarcinKosinski]

There are 38 cohort types in TCGA Study so the maximum data.frame limit for each category is 38 but TCGA not always provide full data and the data.frames' count for selected categories are

pckg.names <- 
  c("RTCGA.clinical", "RTCGA.mutations", "RTCGA.rnaseq",
    "RTCGA.RPPA", "RTCGA.mRNA", "RTCGA.CNV", "RTCGA.miRNASeq",
    "RTCGA.PANCAN12", "RTCGA.methylation")


sapply(pckg.names,
  function(pckg){
   length(data(package = pckg)$results[, "Item"]) 
  }
) -> data.frames.counts

knitr::kable(
  data.frame(
    count = data.frames.counts
  ))

	count
RTCGA.clinical	38
RTCGA.mutations	35
RTCGA.rnaseq	36
RTCGA.RPPA	36
RTCGA.mRNA	13
RTCGA.CNV	37
RTCGA.miRNASeq	38
RTCGA.PANCAN12	5
RTCGA.methylation	17

[vobencha]

OK. Yes, I agree making a package for each type is a good way to organize this.

[MarcinKosinski]

We've chatted with @wchodor and we have prepared such an operation plan:

1. We'll create RTCGA::createTCGA() function that will create ExpressionHub-type package, based on source codes of RTCGA.clinical / RTCGA.mutations / etc.
2. This function will create new packages that we'll submit to ExpressionHub for the newest release date 2016-01-28 and for the future releases in future.
3. I suggest firstly to try to create a draft of a RTCGA::createTCGA() function that will
   1. a) take parameters `createTCGA('date_of_release', 'package_name', 'dataset_name', 'author_name')
   2. b) create only DESCRIPTION file (at the beginning) with full information in this file

@wchodor do you agree this is a good start :)?