This repository is to collect the code for projects to describe and predict the migration of North American hummingbird species for 2004-2013 and into the future. Species include: anna's, allen's, black-chinned, broad-billed, broad-tailed, calliope, costa's, rufous, and ruby-throated hummingbird. Contains code for cleaning eBird data, describing and analyzing species migration paths and timing, exploring environmental and climate correlations with migration, hummingbird physiology, and predicting response to climate change. Generates all results and figures for the paper(s) related to the project. We are working with movebank to automate data annotation with movebank for "non-track" data points. Collaborators on this and related projects include S. R. Supp, F. A. La Sorte, T.A. Cormier, Kevin C. Guay, M. C.W. Lim, G. Bohrer, D. Powers, S. Wethington, S. Goetz, P.A. Beck, and C. H. Graham. Code was written by SRS, TC, KG, and modified from FAL and PAB.
The code and data in this repository allow for the analyses and figures to be fully replicated using eBird data, BBL data, GBIF data, icosahedron map of North America, and remote sensing products.
The project and code in this repository are still under development.
- Data use
- Datasets needed
- Project 1 - Supp et al. 201X. Ecosphere
- Project 2 - eBird + remote sensing
- Extra scripts from exploratory or past analysis
- Query MoveBank (MB)
- PostgreSQL Database Setup
- File Structure
- Submit a Query
Data is currently privately housed, but may be requested for the purposes of replication. If you wish to use the data for additional research, you should contact the Graham lab (email@example.com) and Cornell Lab of Ornithology to request eBird data (http://ebird.org/ebird/data/download), or the corresponding author (firstname.lastname@example.org).
- eBird observation localities with species and date
- eBird list of complete checklists
- eBird effort data by hexagon cell
- icosahedron cell data
- environmental rasters and data from movebank, interpolated to daily values
- Bird Banding Laboratory encounter and recovery dataset
- GBIF data; can be queried using R
This code is available under a BSD 2-Clause License.
R 3.x, R packages
XML, Python 2.7 or greater, Python modules
xml.dom, and files containing functions specific to this code.
The analyses can be replicated by changing the working directory at the top of the file .R to the location on your computer where you have stored the .R, .txt, and .csv files, and by modifying the pathnames for output to where you would like to store the results. Questions about the code should be directed to SRS (email@example.com).
hb-migration.rscript -- cleans up the trait and community datasets, runs descriptive analyses, and outputs figures.
migration-fxns.rscript -- stores functions for subsetting, analyzing, and plotting the data.
BBL_Appendix.rscript -- replicates the figures in Appendix A for the manuscript.
BBLmapcode.rscript -- original script that maps foreign recaptures from Bird Banding Lab data. M. Lim
eBird-effort.rscript -- combines eBird files with icosahedron map to calc effort. Use this one.
eBird-effort-new.rscript -- combines eBird files with icosahedron map
aggregate_hb_checklists.r-- used to initially combine eBird observations, which came as separate month-year files
Project 2 - eBird + remote sensing
agg_hb_checklists_paper2.rscript -- aggregates the monthly checklists FAL sent from eBird, and separates them based on species
hb-gbif.rscript -- loops through the species to identify observed wintering locations in Central America from GBIF data online.
hb-physiological-demand.r-- Calculates standard operative temperature (Te), cost of thermoregulation (Ct) and cost of transport/flight (Cf) from the environmental and geographic variables.
Standard_Operative_Temperature_from_meteorology.R-- GET FROM EXTREME_LIMITS: Functions to calculate standard operative temperature from radiation, temperature and geographic variables for individuals.
Diffuse_fraction_of_solar_radiation.r-- GET FROM EXTREME_LIMITS: Functions to calculate diffuse fraction of solar radiation, solar zenith, and other variables needed for incorporating radiation into standard operative temperature measures.
aggregate_hb_checklists.rscript -- [tc] separates the original queried file into separate files for each species, writes to file. *similar script (newer) by kg:
hb_prep_mb-ebd_tracks.Rscript -- prepares the eBird data for annotation with movebank environmental data
hb_RS_functions.Rscript -- [tc]
hb_alpha_hulls.Rscript -- [kg] calculates alpha hulls for each species, year and week for the day of observation, preceeding week and following week. Generates csv file to be submitted to movebank.
hb_alpha_functions.Rscript -- [kg] functions that are useful for the alpha hull calculation script.
hb_movebank_createXML.pyscript -- [tc] creates xml files for submission to movebank.org for annotation with environmental variables.
hb-submit_request_movebank.rscript -- [tc] submits annotation request to movebank using curl
hb-submit_movebank_request.shscript [LEGACY] -- [tc] generates and submits movebank annotation request using point locations and a request xml generated by
hb_mb_checkStatus_retrieveAnnotation.Rscript [tc] checks the status of movebank queries and downloads data when ready.
hb_movebank_downloadAnnotation_csv.shscript -- [tc] checks status of movebank requests and downloads annotated files when they are available.
hb_mb_combine_datasets.R-- [kg] combine annotated data from movebank into a single dataframe/ CSV (with a column for each variable). Output: one CSV and one RData file per species, abs or prs, and time window (focal, minus, plus).
Extra scripts from exploratory or past analyses
eBird-effort.rscript -- makes maps of total citizen science checklist effort per hex cell for each year, using the effort file downloaded from the eBird portal. Contains daily data. (not needed)
eBird-effort-new.rscript -- processes the eBird effort data (not needed)
hb-centroids.rscript -- current a "junk" script. Is not necessary to run analyses. (not needed)
present_figs.rscript -- generated figures for a presentation in May 2014 (not needed)
Query MoveBank (MB)
PostgreSQL Database Setup
hb_mb_checkStatus_retrieveAnnotation.R scripts use a PostgreSQL database to keep track of submitted movebank queries. The following SQL scripts set up the requsite tables.
You will need to change the following line in
hb_mb_checkStatus_retrieveAnnotation.R to reflect your database information:
con <- dbConnect(drv="PostgreSQL", port="5433",host="fusion.whrc.ad", dbname="kguay")
download_annotations-- MB data is downloaded here
env_request_xmls-- Metadata, used by MB, to identify environmental variables
submit_lists-- List of
status_xmls-- Status files downloaded from MB. This is how we know when a job is done or failed
submit_csv-- CSV files for each query (e.g.
meta-- Metadata about the requests, including (for each species / year / time window) alpha value, number of samples in hull, number of outliers, total samples
submit_lists-- Contains csv files describing which files in
submit_csvto submit to MB
submitted_requests-- XML files used by MB to keep track of submitted requests.
Submit a Query
Change the following lines in
# The following paths need to be changed (path/to/)  source("path/to/hb_RS_functions.R")  xy.csv <- paste0(path/to/movebank/submit_csv/submit_lists/rthu.csv")  xml.csv <- "path/to/movebank/env_request_xmls/submit_lists/submit_06_24_2015.csv"  un <- "tcormier"  pw <- "<password>"  req.outdir <- paste0("path/to/hummingbird/movebank/submitted_requests/")  req.outdir <- paste0("path/to/hummingbird/movebank/downloaded_annotations/")  con <- dbConnect(drv="PostgreSQL", port="5433",host="fusion.whrc.ad", dbname="kguay")
Choose which variables you want to query:
e.g. submit_06_24_2015.csv: path/to/movebank/env_request_xmls/Quality_EVI.xml path/to/movebank/env_request_xmls/lwrf_swrf_t10m_rh20m.xml path/to/movebank/env_request_xmls/MODIS_NPP.xml path/to/movebank/env_request_xmls/SRTM_elev.xml
Choose which species you want to submit:
e.g. rthu only: rthu.csv: # focal path/to/movebank/submit_csv/rthu_abs_movebank_submit_fcl.csv path/to/movebank/submit_csv/rthu_prs_movebank_submit_fcl.csv # +/- 5 days path/to/movebank/submit_csv/rthu_prs_movebank_submit_min_5.csv path/to/movebank/submit_csv/rthu_prs_movebank_submit_pls_5.csv # +/- 10 days path/to/movebank/submit_csv/rthu_prs_movebank_submit_min_10.csv path/to/movebank/submit_csv/rthu_prs_movebank_submit_pls_10.csv # +/- 15 days path/to/movebank/submit_csv/rthu_prs_movebank_submit_min_15.csv path/to/movebank/submit_csv/rthu_prs_movebank_submit_pls_15.csv
Submit MB queries: run
Check the status of the queries: run
Combine individual data products (e.g. NDVI, temp, etc.) into a single dataframe: run