DataSearchDownloadHowto

Tom Kralidis edited this page May 20, 2018 · 24 revisions

Data Search / Download - How to Use

Observation data are made available via the data search / download page, which provides a variety of interactive data search and download options. Data may be selected by type, station, date range and world map location. Selected data may then be downloaded in several different formats for use offline.

Data may be downloaded as individual files as submitted to WOUDC where each file is a record of measurements at a particular station and time period. Multiple file or Bulk data1 access is also available, where all files found as a result of a search can be downloaded at once. Selections of multiple files can be made by Observation Period (temporal) as well as Location (spatial). The files will be packaged in a .zip format file. Note (1): Please note that dataset specific limits are put on bulk data download to maintain quality of service. For larger amounts of bulk data download, please contact us.

1. Select a Dataset The dataset dropdown menu contains downloadable datasets under the following categories:

  • Total Column Ozone
  • Vertical Ozone Profile
  • UV Irradiance
  • Related Data Centres (Data file URLs only)

2. Select a Station Once a dataset is selected, the user can select from a list of associated stations.

3. Select an Instrument Once a station is selected, the user can select from a list of associated instruments. If a station is selected, the instrument selection is based on the station. Otherwise, the instrument list is based on the selected dataset.

4. Select Observation Period and Station Area Users can optionally specify time-based or spatial constraints as part of the search to be provided, allowing for custom search results.

Observation Period There are two ways to adjust the observation period. The graph below automatically updates based on the selection:

  • Slider: there are two sliders that can be adjusted to dynamically change the Observation Period
  • Manually: start and end dates can be manually entered into the labelled boxes

Station Area Map The map represents the geographic area that will be searched. This map view can be adjusted as follows:

  • Panning: click and drag any part of the map
  • Zooming: double-click or using the scroll wheel or by clicking the + or . buttons at the top left of the map. To return to the full map, click the globe icon on the upper left of the map. (Note: zooming the map changes the search area immediately. Only stations within the current map view will be included in the search.)
  • To select an individual station, click the station marker on the map.

5. Data Distribution The graph below the map shows a plot of the total number of observations and the total number of files, limited by the selected observation period and geographic area on the map. The graph changes immediately with changes to the Observation period, Station or Instrument selections.

6. Search Results After selecting a Dataset and Observation period, clicking the search button will bring up a search results list of datasets. (The results list is only updated when the Search button is clicked.) This list can be interacted with in various ways:

  • Show more entries: click on the Show Entries dropdown to change the amount of entries displayed on each page. Additionally, you can cycle through the pages using the arrows or numbers at the bottom
  • Individual raw data files: Each line of the Search Results corresponds to a data file submitted to WOUDC. To download any of these files, click on the link in the Observation Date (UTC) column. You will receive an ExtendedCSV format file exactly as received by WOUDC from the submitter.
  • Download other formats: to download selected data, click the Download. button at the bottom of the page. After choosing a file format, click on the Download records link to begin downloading Note (2): Search results of over 10000 records will be delivered via multiple downloads of 10000 records per download.

7. Download Formats When clicking the Download button, Search results are available in a number of formats produced by WOUDC from submitted data. Unlike clicking on a single Observation date of the Search results to download a single file, using the Download button combines all of the data in the Search results window into one download. (Very lengthy Search results may be compressed into several .zip format files to maintain performance.) The available formats include several ASCII and geospatial formats:

Combined observation data Download from database - CSV: Standard Comma Separated Values file. (This is a text file with commas (,) between fields, that is directly readable by spreadsheet applications and other software systems.) This format combines profile, observation summaries and associated metadata of all the Search results from the WOUDC database. When using Microsoft Excel (tm) to open some WOUDC CSV files (such as sonde data), we recommend turning off the "Wrap Text" option.) The following list shows the CSV content by dataset type, in each case data matching the search criteria (Station name, geographic area etc.) is combined into one file (results.csv) for download:

  • TotalOzone - monthly observations - Daily observation records in monthly files. (Each file contains one month of daily observations.)

  • TotalOzone - daily observations - Observation Summary - Hourly observation records in daily files. (Each file contains one day of hourly observations.)

  • Lidar - Profile Summary - a list of all ozone summary records in chronological order.

  • OzoneSonde - Profile Summary - a file of concatendated ozone sonde flights. The OzoneSonde profile summary contains a header row and one row for each line of results from the Search Results window. Each row contains location coordinates, the fields from the Search Results window and a “data block” field which contains an embedded (nested) CSV of the profile data itself. The embedded CSV of profile data is similar to the submitted profile format and begins with the list of profile elements: Pressure, O3PartialPressure, Temperature, WindSpeed, WindDirection etc., followed by the data values, one row per measurement from the sonde.

  • UmkehrN14 (Level 1.0) - Observations - a list of umkehr 1.0 data files in chronological order. The summary contains a header row and one row for each line of results from the Search Results window. Each row contains location coordinates, the fields from the Search Results window and a “data block” field which contains an embedded (nested) CSV of the UmkehrN14 (N14_VALUES), elements ‘Date,H,W,WLCode,ObsCode,ColumnO3,N_600,N_650,N_700,N_740…’ etc.

  • UmkehrN14 (Level 2.0) - Observations - a list of umkehr 2.0 data files in chronological order. The summary contains a header row and one row for each line of results from the Search Results window. Each row contains location coordinates, the fields from the Search Results window and a “data block” field which contains an embedded (nested) CSV of the UmkehrN142 (C_PROFILE) values, elements ‘Date,H,L,ColumnO3Obs,ColumnO3Retr,Layer10,Layer9,Layer8,…’ etc.

  • Broadband - (this download type is not applicable to this data type.)

  • Multiband - (this download type is not applicable to this data type.)

  • Spectral - Profile Summary - a list of spectral data files in chronological order.

  • NDACC - (this download type is not applicable to this data type.)

Packaged Individual observation files

  • Packaged Extended CSV files in XML wrapper: This is a standard CSV format file with the addition of comments and tags to specify metadata such as station name and location, and different tables within the file. The data files included are the standard file type for submissions of data to WOUDC (as in 6. above,) but this format combines all of the Search results together and packages them in XML and .zip formats to produce a single file for download. (While data can be extracted manually from the downloaded file, various XML tools exist to process XML data programmatically. [XML Processing Example](https://github.com/woudc/woudc /wiki/ProcessingExtendedCSVXMLDownloads) https://github.com/woudc/woudc/wiki/ProcessingExtendedCSVXMLDownloads )
  • Data file URLs: a CSV file containing URL references to each of the selected files. This file format does not contain the data itself, but instead contains a list (URL references) of Extended CSV files which contain the actual data.

Geospatial Formats