Skip to content
This repository has been archived by the owner on Dec 12, 2023. It is now read-only.

Latest commit

 

History

History
628 lines (530 loc) · 25.3 KB

NOTES.rst

File metadata and controls

628 lines (530 loc) · 25.3 KB
Raw

LabCAS Backend

Installation generally has the following

  • XML-RPC File Manager on port 9000
  • XML-RPC Workflow Manager on port 9001
  • XML-RPC Resource Manager on port 9002
  • Apache Tomcat on port 8080 (shutdown port 8005, AJP on port 8009) with webapps: - fmprod
  • SOLR (in its included Jetty) on port 8983

There is also Apache HTTPD on ports 80/443 but essentially doing nothing.

ssh -L 9000:localhost:9000 -L 9001:localhost:9001 -L 9002:localhost:9002 -L 8080:localhost:8080 -L 8983:localhost:8983 labcas-dev.jpl.nasa.gov

Docker

To build:

env LABCAS_UI_VERSION=1.2.3 docker-compose build

This will create an image edrn.labcas.ui:1.2.3. If you ommit the LABCAS_UI_VERSION, the default tag of "latest" will be used instead.

To run:

docker-compose up

Luca

FYI if you ever need the labcas-backend python examples again, I refactored them in this little package:

https://github.com/EDRN/labcas-backend/tree/master/common/src/main/python/gov/nasa/jpl/edrn/labcas

---

Hi Sean,
here’s a proposal for how the Labcas backend could support your new improved Labcas front-end pages:
>>> "My Data” page:
o) Each workflow run by the user is assigned a “dataset name” and a “version”. These could be passed as part of the job metadata,
or default to some sensible values. All product files generated by that job will be assigned the same “dataset name” and “version"
  1. additionally, the Labcas backend will automatically generate a “status” and a “last update” field, as well as the “size” for each file in the dataset

o) Once the job completes and all the metadata is in Solr, the Labcas front-end will have no problem in retrieving all the dataset information from Solr. In particular, I will provide you with example queries to:

  • retrieve all datasets for a given product type
  • retrieve summary information for each dataset (includes dataset name, number of files, total size, version, last update)
  • retrieve all download URLs and sizes for all files in the dataset
>>> “Dataset Detail” page
o The Files table on the right can be built by querying Solr as described above

o I am not sure we want to enable “Delete checked Files” - maybe the dataset should be deleted as a whole or nothing ?

o We need to discuss where the information on the left is coming from:
  • inserted by the user at time of submission ?
  • already existing in a database ?

Depending on the answer, we will have to either: a) pass all that information as job metadata and store it in Solr as well b) pass a reference to some unique dataset information identifier in the database c) let the Labcas front-end retrieve that information independently, using the same “dataset name"

If this sounds like a good plan, I can try to make it happen next week (well, at least the first part).

thanks, Luca

P.S.: I am not worried about hierarchical datasets for now. When we have a real use case, we can implement it using one of the suggestions made by Chris.

On Sep 10, 2015, at 1:23 PM, Sean Kelly <sean.kelly@jpl.nasa.gov> wrote:

> Superb! > > Thanks Chris. > --k > >> On 2015-09-10, at 11:55 AM, Mattmann, Chris A (3980) <chris.a.mattmann@jpl.nasa.gov> wrote: >> >> Yeah this can be supported in OODT in various ways. Product Types >> can easily just have metadata that links to other product types, >> and then you can build knowledge on top that just uses IDs to join >> things. >> >> OODT-40 talked about building an API for this, but no one really >> did Another option is just to use hierarchical products, which >> also will work. >> >> Cheers, >> Chris >> >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Chris Mattmann, Ph.D. >> Chief Architect >> Instrument Software and Science Data Systems Section (398) >> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA >> Office: 168-519, Mailstop: 168-527 >> Email: chris.a.mattmann@nasa.gov >> WWW: http://sunset.usc.edu/~mattmann/ >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> Adjunct Associate Professor, Computer Science Department >> University of Southern California, Los Angeles, CA 90089 USA >> ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++ >> >> >> >> >> >> -----Original Message----- >> From: "Cinquini, Luca (398G)" <Luca.Cinquini@jpl.nasa.gov> >> Date: Thursday, September 10, 2015 at 7:21 AM >> To: "Kelly, Sean C (398G-Affiliate)" <sean.c.kelly@jpl.nasa.gov> >> Cc: "Crichton, Daniel J (3902)" <daniel.j.crichton@jpl.nasa.gov>, edrn-ic >> <edrn-ic@jpl.nasa.gov> >> Subject: Re: [edrn-ic] Updated LabCAS graphics >> >>> >>> >>> Hi Sean, >>> I think what we call “dataset" maps to an OODT product. >>> >>> From my limited knowledge, OODT has the concept of “hierarchical >>> product”, i.e. a product that can contain many files, organized in a >>> directory tree. But, products cannot contain other products. >>> Maybe others can correct me. If I am right, we’ll have to think about >>> this… >>> >>> >>> thanks, Luca >>> >>> On Sep 10, 2015, at 7:55 AM, Sean Kelly <sean.kelly@jpl.nasa.gov> wrote: >>> >>> >>> Yes indeed. Luca, did you see my questions earlier? Can OODT filemgr >>> handle "composite" datasets, that is: a dataset that contains additional >>> datasets? >>> >>> >>> Thanks >>> --k >>> >>> >>> On 2015-09-09, at 10:24 PM, Crichton, Daniel J (3902) >>> <daniel.j.crichton@jpl.nasa.gov> wrote: >>> >>> Hi Sean >>> >>> >>> We didn't manage to get a staff meeting scheduled this week, but perhaps >>> we can assemble a group to talk to this Friday? Luca are you available? >>> >>> >>> For a lot of our data sets, we do want to handle product level metadata >>> and even may want to do product level search. This is something for PDS, >>> for example, that has become very important. With our EDRN data sets, a >>> lot of the files are very >>> unique so I think we need to annotate each one. We also may need to >>> allow specialization for different type of data. >>> >>> >>> Thanks, >>> Dan >>> >>> >>> >>> >>> >>> >>> From: Sean Kelly <sean.kelly@jpl.nasa.gov> >>> Date: Tuesday, September 8, 2015 12:06 PM >>> To: Dan Crichton <Daniel.J.Crichton@jpl.nasa.gov> >>> Cc: edrn-ic <edrn-ic@jpl.nasa.gov> >>> Subject: Re: [edrn-ic] Updated LabCAS graphics >>> >>> >>> >>> >>> 1. Is the metadata on the left mapped to eCAS? >>> >>> >>> >>> >>> >>> >>> That's a good idea. >>> >>> >>> >>> 2. Can we handle nested sets? I believe Dave Tabb had an example where a >>> set was in a set. >>> >>> >>> >>> >>> >>> >>> I'm not sure. LUCA: Can OODT filemgr handle nested datasets? >>> >>> >>> 3. How do we handle product metadata? >>> >>> >>> >>> >>> >>> >>> I'm not sure. >>> >>> >>> 4. Download all may be a pretty common occurrence for small data sets. I >>> think we have that in LabCAS today >>> >>> >>> >>> >>> >>> >>> I'll make sure we have a "download all" button. >>> >>> >>> 5. How can we share with other groups. I think LabCAS today allowed >>> folks to indicate someone else to share with. >>> >>> >>> >>> >>> >>> >>> Good point. Let me mock up some "Sharing" settings. >>> >>> >>> 6. On the first page, I'm thinking a lot of folks will just login to grab >>> data shared with them. They won't be processing, etc. Might want to >>> show who the PI is for the data set. >>> >>> >>> >>> >>> >>> >>> Gotcha. OK the summary list should show "My Data" and "Data Shared with >>> Me" — and a PI column in both. >>> >>> >>> Thanks for the feedback! >>> --k >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >>> >

I deployed the new LabCAS back-end functionality on edrn-frontend. This is the sequence of calls that the labcas-UI needs to execute at startup to obtain all the information it needs to build the UI. You’ll also be happy to know that I reduced the number of needed calls from 4 to 3….
  1. Find all the events triggering workflows:

./wmgr-client -op -events -url http://edrn-frontend:9001 Event: [name=labcas-test] Event: [name=biomarker-discovery] Event: [name=biomarker-discovery-0] Event: [name=biomarker-discovery-2]

(NOTE: for now the only workflow that can self-describe itself is “labcas-test”)

  1. For each event, find the workflow it executes:

./wmgr-client -op --getWorkflowsByEvent --eventName labcas-test -url http://edrn-frontend:9001 Workflow: [id=urn:edrn:LabcasTestWorkflow, name=LabcasTestWorkflow, numTasks=2, taskIds=urn:edrn:LabcasTestInit, urn:edrn:LabcasTestTask]

(NOTE: you are getting the task ids right away - this will save you one call…)

  1. Retrieve the first task to obtain all the configuration needed to build the UI:

./wmgr-client -op --getTaskById --id urn:edrn:LabcasTestInit -url http://edrn-frontend:9001 Task: [id=urn:edrn:LabcasTestInit, name=LabcasTestInit, order=-1, class=org.apache.oodt.cas.workflow.structs.WorkflowTask, numConditions=0, requiredMetadataFields=[experiment, species], configuration={input.species.type=string, input.experiment.title=Experiment, input.experiment.type=integer, input.species.title=Species, input.species.value=sharks, input.experiment.min=1, output.product.type=urn:edrn:LabcasTestFile, input.experiment.description=The experiment sequential number, input.experiment.max=99, input.species.description=The species that was tested}]

(NOTE: you are getting both the required fields, and for each fields a whole bunch of information)

Then, after the user makes a selection for all the required parameters, the labcas-UI can:

  1. Execute the workflow:
./wmgr-client --url http://edrn-frontend:9001 --operation --sendEvent --eventName labcas-test --metaData --key experiment 11 --key species snakes

Sending event 'labcas-test'… SUCCESS

  1. Query results from Solr (of this particular type, sorting by date descending order):

http://edrn-frontend.jpl.nasa.gov:8080/solr/oodt-fm/select?q=*%3A*&fq=+CAS.ProductTypeName%3DLabcasTestFile&sort=CAS.ProductReceivedTime+DESC&wt=json&indent=true

Please let me know of any problems… hopefully no problems….

thanks, Luca

Hi Sean,
I deployed the latest version of labcas-backend on biomarker if you want to start - I will deploy the same on the new server “labcas-dev” as soon as it becomes available.

This version supports both workflows:

  1. “labcas-upload” to publish a dataset
  2. “labcas-update” to update the metadata of the latest dataset version

Details on 1) were contained in a previous email. Details on 2) follow. Please let me know of any questions - looking forward to work with you into having an end-to-end system.

thanks, Luca

To update the metadata for an already published dataset, simply creates a file with new metadata content in $LABCAS_STAGING/<dataset name>/DatasetMetadata.xml

Each metadata field contained in the file "DatasetMetadata.xml" will be updated, completely replacing all previous values for that field. To insert multiple values for the same field, simply specify multiple "<keyval>" elements. To completely remove all values for a field, leave the "<val>" empty.

Example of DatasetMetadata.xml:

<?xml version="1.0" encoding="UTF-8" standalone="no"?> <cas:metadata xmlns:cas="http://oodt.jpl.nasa.gov/1.0/cas">

<keyval type="vector">
<key>PI</key> <val></val>

</keyval> <keyval type="vector">

<key>Institution</key> <val>Darthmouth</val>

</keyval> <keyval type="vector">

<key>Institution</key> <val>Children Hospital</val>

</keyval>

</cas:metadata>

will cause: o the field "PI" to be removed from all macthing records o the field "Institution" to be assigned the two possible values "Darthmouth" and "Children Hospital”

Note that the metadata will be replaced ONLY for the latest dataset version - all other versions will be unaffected.

To trigger the workflow:

  1. from the command line:

./wmgr-client --url http://localhost:9001 --operation --sendEvent --eventName labcas-update --metaData --key Dataset mydata

  1. example XML-RPC in Python (see also https://github.com/EDRN/labcas-backend/blob/master/examples/labcas-xmlrpc-example-client.py)

wInstId = server.workflowmgr.executeDynamicWorkflow( ['urn:edrn:LabcasUpdateTask'], { 'Dataset':'mydata' } )

Again please let me know of any questions...

Thanks Chris, that would be helpful too.

As an update, I already installed it, since it was extremely easy. Now, once a workflow is run and the products are ingested into the file manager, you can retrieve them by product_id, for example:

http://edrn-frontend.jpl.nasa.gov:8080/fmprod/data?productID=499e9468-a79e-4145-b563-4954fb63007f

Sean, please let mw know if this is ok

thanks, L

On Aug 30, 2015, at 6:01 PM, Mattmann, Chris A (3980) <chris.a.mattmann@jpl.nasa.gov> wrote:

> The product server also delivers the REST rdf feed... > > Sent from my iPhone > >> On Aug 30, 2015, at 4:59 PM, Cinquini, Luca (398G) <Luca.Cinquini@jpl.nasa.gov> wrote: >> >> Hi Sean, >> first of all happy birthday! >> >> Second of all, I am looking at making the OODT product server part of the labcas-backend package, i.e. it would be installed together with the OODT workflow manager, file manager, etc., >> to allow download of files from the File Manager. >> >> Do you have any knowledge that this option should not be used - for example, for performance reasons ? Other possibilities include installing an Apache server, or a Thredds Data Server, but I thought that the OODT Product Server would be the most logical choice considering all the other OODT components we are already installing. >> >> Please let me know - if you don’t have anything against it, I’ll start working on it. >> >> thanks, Luca >> >>

Hi Sean,
I installed the labcas-backend stack on both “labcas” and “labcas-dev”. I think I sent you instructions on how to submit and monitor uploads already, plus how to query the Solr index,

but to re-iterate perhaps the best examples to start from when programming in python are listed here:

https://github.com/EDRN/labcas-backend/tree/master/examples

Please let me know what I can do for you - I can talk any time, and we should definitely talk sometimes next week to see how the back-end should be augmented for the demo in Mid-October.

thanks a lot, Luca

Hi Sean,
I have something deployed on biomarker that you can start using to test publication of uploaded files. There’s probably much refinement to do,

but this should get you started. Next week I’ll try to work on editing existing metadata.

Steps:

  1. login to biomarker, ssh edrn@localhost

2) drop your files in a directory of the form: /usr/local/labcas/staging/<dataset name> (for example, <dataset name>=mydata)

  1. trigger the “labcas-upload” workflow, for example from the command line:

cd $LABCAS_HOME/cas-workflow/bin ./wmgr-client --url http://localhost:9001 --operation --sendEvent --eventName labcas-upload --metaData --key Dataset mydata

I also wrote two simple python scripts to show hot to submit, monitor and query products from python:

https://github.com/EDRN/labcas-backend/blob/master/examples/labcas-xmlrpc-example-client.py

https://github.com/EDRN/labcas-backend/blob/master/examples/labcas-solrpy-example-client.py (this script requires solrpy installed - you can run it from your laptop or ask Paul to install solrpy on biomarker).

In the next days/weeks we can improve on what’s there.

thanks, L

--

'http://localhost:8983/solr/select?fq=ProtocolId%3A138&version=2.2&q=%2A%3A%2A&start=0&wt=standard&fl=%2A%2Cscore'

SOLR reply:

[
{
u'_version_': 1528439907579592704L, u'CAS.ProductId': u'FHCRCHanashAnnexinLamr.1.(By Batch)ANXA1(FR-00-84).pdf', u'CAS.ProductName': u'(By Batch)ANXA1(FR-00-84).pdf', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 10, 18, 3, tzinfo=<solr.core.UTC>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:FHCRCHanashAnnexinLamr', u'CAS.ProductTypeName': u'FHCRCHanashAnnexinLamr' u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/FHCRCHanashAnnexinLamr/1/(By%20Batch)ANXA1(FR-00-84).pdf'], u'CAS.ReferenceFileSize': [22248L], u'CAS.ReferenceMimeType': [u'application/pdf'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/FHCRCHanashAnnexinLamr/(By%20Batch)ANXA1(FR-00-84).pdf'], u'DatasetId': u'FHCRCHanashAnnexinLamr', u'id': u'FHCRCHanashAnnexinLamr.1.(By Batch)ANXA1(FR-00-84).pdf', u'LeadPI': [u'Samir Hanash'], u'ParentDatasetId': [u'EcasProduct'], u'score': 1.0, u'Version': 1,
}, {
u'_version_': 1528439907400286208L, u'CAS.ProductId': u'FHCRCHanashAnnexinLamr.1.(By Batch)14-3-3.pdf', u'CAS.ProductName': u'(By Batch)14-3-3.pdf', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 10, 18, 3, tzinfo=<solr.core.UTC object at 0x10927b610>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:FHCRCHanashAnnexinLamr', u'CAS.ProductTypeName': u'FHCRCHanashAnnexinLamr' u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/FHCRCHanashAnnexinLamr/1/(By%20Batch)14-3-3.pdf'], u'CAS.ReferenceFileSize': [20757L], u'CAS.ReferenceMimeType': [u'application/pdf'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/FHCRCHanashAnnexinLamr/(By%20Batch)14-3-3.pdf'], u'DatasetId': u'FHCRCHanashAnnexinLamr', u'id': u'FHCRCHanashAnnexinLamr.1.(By Batch)14-3-3.pdf', u'LeadPI': [u'Samir Hanash'], u'ParentDatasetId': [u'EcasProduct'], u'score': 1.0, u'Version': 1,
}, {
u'_version_': 1528439908116463616L, u'CAS.ProductId': u'FHCRCHanashAnnexinLamr.1.(By Batch)PGP9.5.pdf', u'CAS.ProductName': u'(By Batch)PGP9.5.pdf', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 10, 18, 4, tzinfo=<solr.core.UTC object at 0x10927b610>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:FHCRCHanashAnnexinLamr', u'CAS.ProductTypeName': u'FHCRCHanashAnnexinLamr' u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/FHCRCHanashAnnexinLamr/1/(By%20Batch)PGP9.5.pdf'], u'CAS.ReferenceFileSize': [21177L], u'CAS.ReferenceMimeType': [u'application/pdf'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/FHCRCHanashAnnexinLamr/(By%20Batch)PGP9.5.pdf'], u'DatasetId': u'FHCRCHanashAnnexinLamr', u'id': u'FHCRCHanashAnnexinLamr.1.(By Batch)PGP9.5.pdf', u'LeadPI': [u'Samir Hanash'], u'ParentDatasetId': [u'EcasProduct'], u'score': 1.0, u'Version': 1,
}, {
u'_version_': 1528439907946594304L, u'CAS.ProductId': u'FHCRCHanashAnnexinLamr.1.(By Batch)LAMR1.pdf', u'CAS.ProductName': u'(By Batch)LAMR1.pdf', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 10, 18, 4, tzinfo=<solr.core.UTC object at 0x10927b610>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:FHCRCHanashAnnexinLamr', u'CAS.ProductTypeName': u'FHCRCHanashAnnexinLamr' u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/FHCRCHanashAnnexinLamr/1/(By%20Batch)LAMR1.pdf'], u'CAS.ReferenceFileSize': [21446L], u'CAS.ReferenceMimeType': [u'application/pdf'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/FHCRCHanashAnnexinLamr/(By%20Batch)LAMR1.pdf'], u'DatasetId': u'FHCRCHanashAnnexinLamr', u'id': u'FHCRCHanashAnnexinLamr.1.(By Batch)LAMR1.pdf', u'LeadPI': [u'Samir Hanash'], u'ParentDatasetId': [u'EcasProduct'], u'score': 1.0, u'Version': 1,
}, {
u'_version_': 1528439907764142080L, u'CAS.ProductId': u'FHCRCHanashAnnexinLamr.1.(By Batch)DJ-1.pdf', u'CAS.ProductName': u'(By Batch)DJ-1.pdf', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 10, 18, 4, tzinfo=<solr.core.UTC object at 0x10927b610>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:FHCRCHanashAnnexinLamr', u'CAS.ProductTypeName': u'FHCRCHanashAnnexinLamr' u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/FHCRCHanashAnnexinLamr/1/(By%20Batch)DJ-1.pdf'], u'CAS.ReferenceFileSize': [21821L], u'CAS.ReferenceMimeType': [u'application/pdf'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/FHCRCHanashAnnexinLamr/(By%20Batch)DJ-1.pdf'], u'DatasetId': u'FHCRCHanashAnnexinLamr', u'id': u'FHCRCHanashAnnexinLamr.1.(By Batch)DJ-1.pdf', u'LeadPI': [u'Samir Hanash'], u'ParentDatasetId': [u'EcasProduct'], u'score': 1.0, u'Version': 1,
}, {
u'_version_': 1528439908315693056L, u'CAS.ProductId': u'FHCRCHanashAnnexinLamr.1.CARET lung cancer.xls', u'CAS.ProductName': u'CARET lung cancer.xls', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 10, 18, 4, tzinfo=<solr.core.UTC object at 0x10927b610>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:FHCRCHanashAnnexinLamr', u'CAS.ProductTypeName': u'FHCRCHanashAnnexinLamr'} u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/FHCRCHanashAnnexinLamr/1/CARET%20lung%20cancer.xls'], u'CAS.ReferenceFileSize': [57856L], u'CAS.ReferenceMimeType': [u'application/vnd.ms-excel'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/FHCRCHanashAnnexinLamr/CARET%20lung%20cancer.xls'], u'DatasetId': u'FHCRCHanashAnnexinLamr', u'id': u'FHCRCHanashAnnexinLamr.1.CARET lung cancer.xls', u'LeadPI': [u'Samir Hanash'], u'ParentDatasetId': [u'EcasProduct'], u'score': 1.0, u'Version': 1,

]

--

[
{
u'_version_': 1528437820944809984L, u'CAS.ProductId': u'GSTP1_Methylation.1.Methylation Data Codebook(2).docx', u'CAS.ProductName': u'Methylation Data Codebook(2).docx', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 9, 44, 53, tzinfo=<solr.core.UTC object at 0x10a615650>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:GSTP1_Methylation', u'CAS.ProductTypeName': u'GSTP1_Methylation', u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/GSTP1_Methylation/1/Methylation%20Data%20Codebook(2).docx'], u'CAS.ReferenceFileSize': [17225L], u'CAS.ReferenceMimeType': [u'application/vnd.openxmlformats-officedocument.wordprocessingml.document'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/GSTP1_Methylation/Methylation%20Data%20Codebook(2).docx'], u'DatasetId': u'GSTP1_Methylation' u'DataStage': [u'Processed'], u'DateProductFrozen': [u'NA'], u'id': u'GSTP1_Methylation.1.Methylation Data Codebook(2).docx', u'InstrumentDetails': [u'ntt7900HT sequence detector (Applied Biosystems). nttSDS 2.4 software for methylation analysis. nttAll statistical analysis was done with SAS, version 9.2.ntt'], u'InstrumentId': [u'NA'], u'LeadPI': [u'David Sidransky'], u'MethodDetails': [u'NA'], u'ParentDatasetId': [u'EcasProduct'], u'ProductDescription': [u'Methylation Data Cookbok (including Variables Definition)'], u'ProductTitle': [u'Methylation Data Codebook'], u'score': 1.0, u'SpecimenId': [u'Tissue paraffin block'], u'SpecimenType': [u'Tissue paraffin block'], u'Version': 1,
}, {
u'_version_': 1528437821179691008L, u'CAS.ProductId': u'GSTP1_Methylation.1.methylation_data_2.sas7bdat', u'CAS.ProductName': u'methylation_data_2.sas7bdat', u'CAS.ProductReceivedTime': datetime.datetime(2016, 3, 10, 9, 44, 54, tzinfo=<solr.core.UTC object at 0x10a615650>), u'CAS.ProductStructure': u'Flat', u'CAS.ProductTransferStatus': u'RECEIVED', u'CAS.ProductTypeId': u'urn:edrn:GSTP1_Methylation', u'CAS.ProductTypeName': u'GSTP1_Methylation', u'CAS.ReferenceDatastore': [u'file:/usr/local/labcas/backend/archive/GSTP1_Methylation/1/methylation_data_2.sas7bdat'], u'CAS.ReferenceFileSize': [196608L], u'CAS.ReferenceMimeType': [u'application/x-sas-data'], u'CAS.ReferenceOriginal': [u'file:/usr/local/labcas/backend/staging/GSTP1_Methylation/methylation_data_2.sas7bdat'], u'DatasetId': u'GSTP1_Methylation' u'DataStage': [u'Processed'], u'DateProductFrozen': [u'NA'], u'id': u'GSTP1_Methylation.1.methylation_data_2.sas7bdat', u'InstrumentDetails': [u'ntt7900HT sequence detector (Applied Biosystems). nttSDS 2.4 software for methylation analysis. nttAll statistical analysis was done with SAS, version 9.2.ntt'], u'InstrumentId': [u'NA'], u'LeadPI': [u'David Sidransky'], u'MethodDetails': [u'NA'], u'ParentDatasetId': [u'EcasProduct'], u'ProductDescription': [u'Methylation SAS data'], u'ProductTitle': [u'Methylation SAS data'], u'score': 1.0, u'SpecimenId': [u'Tissue paraffin block'], u'SpecimenType': [u'Tissue paraffin block'], u'Version': 1,

}

]