Upload 2016 data from KSU to terraref/bety

[dlebauer]

Data from KSU 2016 field season is in here:

The most straightforward way to get these data into BETYdb will be to use the API. The web interface is also handy, especially for information like defining traits and agronomic managements where there are only a handful of records. (you could also use SQL but the API hides a lot of complexity).

If you would prefer to upload to BMS, then we can work on the BMS-->BETYdb importer. This assumes BMS exports BRAPI, so this importer would also work with FieldBook and any other data source that exports BRAPI.

Completion Criteria

• define any new plots, genotypes, traits and methods. These can be done in a spreadsheet or in the web interface.
• Prepare the trait csv files for upload following the API specifications (and ask @gsrohde for help).
• upload the data to google docs or directly to BETYdb
  • fieldbook data
  • plot level summaries HTP platforms
• check scripts into a new github repository;
  • follow up: consider making an extractor that automates FieldBook --> BETYdb imports or FieldBook to BRAPI and BRAPI --> BETYdb

References

Templates for the data we need are here: https://docs.google.com/spreadsheets/d/1WmWyVhGfKHEL-i9E2JOEt1txiAgTL4n3lbyQnqsUbNA/edit#gid=1497549722);

There are some examples of measurements we have uploaded from MAC in the google drive folder (Maricopa Experiments/Field Seasons 1 and 2/Data for uploading/). This one is fairly comprehensive

[dlebauer]

@gpmorrisksu what is the plan?

[gpmorrisksu]

Hi David, Zhenbin has already sent the 2016 manual phenotyping data, no? If the question is fieldbook to BrAPI format conversion, I think that better addressed in your group. I think our responsibility lies in producing the data and analyzing G-to-P. Best, Geoff

…

------------------------------------ Geoff Morris, Assist. Professor Kansas State Univ., Agronomy 3004 Throckmorton<x-apple-data-detectors://6/0> Plant Sci. Manhattan KS, 66506<x-apple-data-detectors://6/0> E-mail: gpmorris@k-state.edu<mailto:gpmorris@k-state.edu> Web:http://www.morrislab.org<http://www.morrislab.org/> Office: 785-532-3397<tel:785-532-3397> Cell: 312-909-1330<tel:312-909-1330> Skype/Google: morris.geoff.p

Sent from my mobile device On Mar 8, 2017, at 10:47 PM, David LeBauer <notifications@github.com<mailto:notifications@github.com>> wrote: @gpmorrisksu<https://github.com/gpmorrisksu> what is the plan? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#85 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/ATLjZBi3yVk7_oZ-54cPbCHLNUQlrVr3ks5rjy_vgaJpZM4L0JSX>.

[dlebauer]

@gpmorrisksu sorry if I was not clear. When entering data, we need to make sure that we have all of the information that would be required for someone to make use of the primary data.

This includes the definitions of phenotypes and methods used to collect them, the agronomic managements (planting date and density, fertilization, tillage, experimental design, plot layouts), and the genotypes, used,

My groups responsibility is to provide a place to store this data and metadata and make it available. We can help with the entire process. But it is not our responsibility to interpret any set of tables that are emailed to us.

My understanding was that fieldbook was already capable of exporting to the BRAPI format, which includes all of this information. Supporting import (and export) of data via community supported formats such as the BRAPI api is our responsibility. So if it is easier to supply your information in that format than via our web pages and bulk upload interface or API then that would also work.

[dlebauer]

@zhenbinHU can you please update this issue with a list of the steps that you have accomplished with respect to preparing data and metadata.

[max-zilla]

@zhenbinHU please check / update completion criteria and let us know if you need input

[zhenbinHU]

@dlebauer
PG::RaiseException: ERROR: The value of cultivar_id (6000000456) is not consistent with the value 6000000937 specified for site_id 6000004483. : INSERT INTO "traits" ("access_level", "checked", "citation_id", "created_at", "cultivar_id", "date", "date_day", "date_month", "date_year", "dateloc", "entity_id", "mean", "method_id", "n", "notes", "site_id", "specie_id", "stat", "statname", "time", "time_hour", "time_minute", "timeloc", "treatment_id", "updated_at", "user_id", "variable_id") VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27) RETURNING "id"

After I click the insert data for uploading data to betydb, I have this error.

[NewcombMaria]

Is this issue the same topic as #119 (Upload hand measured traits from KSU 2016). There might be separate conversations going on about betydb data upload errors. See comment by @gsrohde in issue 119.

[zhenbinHU]

Thanks, @NewcombMaria Yes, I just remove the data (five plots) with errors from the uploading data.

[gsrohde]

@NewcombMaria , @zhenbinHU In this case, it seems the error does have to do with the sites_cultivars table. The sites_cultivars table says that for site 6000004483 ("Ashland Bottoms KSU 2016 Season Range 25 Pass 22"), the cultivar there is 6000000937 ("PI569459"), but the upload file specifies cultivar 6000000456 ("Blade"). If you believe the sites_cultivars table is correct, you can simply remove the cultivar column from the upload file and the correct cultivar will be supplied automatically.