Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add maestro + platform mappings #18

Merged
merged 23 commits into from
Apr 6, 2020
Merged

add maestro + platform mappings #18

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
753b5c6
add maestro + platform mappings
rosibaj Mar 26, 2020
e620254
typos
rosibaj Mar 26, 2020
e6dba6c
add settings/autocomplete/types
rosibaj Mar 26, 2020
5ac837b
brackets
rosibaj Mar 26, 2020
1e194c0
add s to mapping
rosibaj Mar 26, 2020
c0fa0ff
Update maestro_file_centric.json
blabadi Mar 27, 2020
e800a39
Update platform_file_centric.json
blabadi Mar 27, 2020
ee1deb2
correct date type on dates
rosibaj Mar 30, 2020
4bd9316
Merge remote-tracking branch 'origin/add_es_mappings' into add_es_map…
rosibaj Mar 30, 2020
1bb15fb
correct md5_sum
rosibaj Mar 30, 2020
0f1a9f7
fix types
rosibaj Mar 30, 2020
6ff4263
remove dynamic - do group by workflow
rosibaj Mar 30, 2020
419744e
update analyzers
blabadi Apr 1, 2020
0336b41
add analyzer to platform file mapping
blabadi Apr 3, 2020
466bc11
remove sequencing fields from mapping
rosibaj Apr 3, 2020
ba209bd
remove/add workflow field
rosibaj Apr 3, 2020
ce2a407
actually commit removed workflow
rosibaj Apr 3, 2020
84b3c30
rename workflow fields
rosibaj Apr 6, 2020
e1f798a
add additional autocompletes
rosibaj Apr 6, 2020
d92b54f
Update maestro_file_centric.json
blabadi Apr 6, 2020
804d7a8
Update platform_file_centric.json
blabadi Apr 6, 2020
45cbef1
- remove PD id as it one snot have one _yet_
rosibaj Apr 6, 2020
abbeb3f
Merge remote-tracking branch 'origin/add_es_mappings' into add_es_map…
rosibaj Apr 6, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 189 additions & 0 deletions mappings/maestro_file_centric.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,189 @@
{
"aliases": null,
"settings": {
"index.stuff": null
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
},
"mapping": {
"dynamic": false,
"date_detection": false,
"properties": {
"file_autocomplete": {
"fields": {
"analyzed": null,
"lowercase": null
},
"type": "keyword"
},
"study_id": {
"type": "keyword"
},
"object_id": {
"type": "keyword"
},
"file_type": {
"type": "keyword"
},
"data_type": {
"type": "keyword"
},
"file_access": {
"type": "keyword"
},
"analysis": {
"properties": {
"analysis_id": {
"type": "keyword"
},
"analysis_type": {
"type": "keyword"
},
"analysis_version": {
"type": "integer"
},
"experiment": {
"type": "object",
"properties": {
"platform": {
"type": "keyword"
},
"platform_model": {
"type": "keyword"
},
"sequencing_date": {
"type": "keyword"
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
},
"library_name": {
"type": "keyword"
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
},
"library_strategy": {
"type": "keyword"
},
"sequencing_center": {
"type": "keyword"
}
}
}
}
},
"dynamic": {
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
"properties": {
"workflow_name": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

workflow in SONG payload contains fields: name, short_name, version, run_id and inputs. Do we want all of them here or just selective ones?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

shouldn't this be under analysis not file ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and will we have workflow section in the analysis dynamic schema ?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah, workflow under analysis makes more sense

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I moved workflow under analysis, that being said I think there should be a correspondence between song schema and this mapping structure for this to work .. i.e. song analysis schema should have workflow object in the payload

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it does. Well mostly, except that sequencing_experiment doesn't have workflow in it since it's not produced from a workflow. I think this should be still OK, basically workfow is optional under analysis, and ES is able to handle optional fields

"type": "keyword"
},
"workflow_type": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does this come from?

"type": "keyword"
},
"analysis_software": {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

where does this come from?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some of these are based on draft mapping you had a year or less ago, so some fields are outdated

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@junjun-zhang after reviewin ICGC and GDC, this is something that is currently missing from SONG payload. while we have the workflow name, we dont include teh software used (like BWA MEM for alignment).

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, since SONG analysis type schemas are more fleshed out, should be good to take what's currently in SONG

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, @rosibaj we discussed about this in bioinfo meetings, decided not to include tool level information in the SONG workflow section

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@christinayung do we want the tool level information on the platform as a searchable feature?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since there can be many tools with changing version, the list may be hard to maintain. I'd suggest storing the workflow version so that the tools can be inspected at github in details.

"type": "keyword"
}
}
},
"file": {
"properties": {
"id": {
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
"type": "keyword"
},
"size": {
"type": "integer"
},
"md5sum": {
"type": "keyword"
},
"name": {
"type": "keyword"
}
}
},
"index_file": {
"properties": {
"object_id": {
"=type": "keyword"
},
"format": {
"type": "keyword"
},
"md5_sum": {
blabadi marked this conversation as resolved.
Show resolved Hide resolved
"type": "keyword"
},
"name": {
"type": "keyword"
},
"size": {
"type": "integer"
}
}
},
"donor": {
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
"type": "nested",
"properties": {
"id": {
"type": "keyword"
},
"submitter_donor_id": {
"type": "keyword"
},
"gender": {
"type": "keyword"
},
"specimen": {
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
"type": "nested",
"properties": {
"id": {
"type": "keyword"
},
"submitter_specimen_id": {
"type": "keyword"
},
"tumour_normal_designation": {
"type": "keyword"
},
christinayung marked this conversation as resolved.
Show resolved Hide resolved
"specimen_tissue_source": {
"type": "keyword"
},
"specimen_type": {
"type": "keyword"
},
"samples": {
"type": "nested",
"properties": {
"id": {
"type": "keyword"
},
"submitted_id": {
"type": "keyword"
},
"sample_type": {
"type": "keyword"
},
"matched_normal_submitter_sample_id": {
"type": "keyword"
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
}
}
}
}
}
}
},
"repositories": {
"type": "nested",
"properties": {
"code": {
"type": "keyword"
},
"name": {
"type": "keyword"
},
"organization": {
"type": "keyword"
},
"country": {
"type": "keyword"
},
"url": {
rosibaj marked this conversation as resolved.
Show resolved Hide resolved
"type": "keyword"
}
}
}
}
}
}
Loading