cmg_ts2cf_no_stationdim.ipynb

{
 "metadata": {
  "name": ""
 },
 "nbformat": 3,
 "nbformat_minor": 0,
 "worksheets": [
  {
   "cells": [
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "#Convert existing time series NetCDF files in 4D (T,Z,Y,X) \"grid\" format to CF-1.6 format"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "import glob\n",
      "import os\n",
      "import netCDF4\n",
      "import StringIO\n",
      "import pandas as pd\n",
      "import re"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 70
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "projs = '''\n",
      "ARGO_MERCHANT,B. Butman,Argo Merchant Experiment,A moored array deployed after the ARGO MERCHANT ran aground onNantucket Shoals designed to help understand the fate of the spilled oil.\n",
      "BUZZ_BAY,B. Butman,Currents and Sediment Transport in Buzzards Bay,Investigation of the near-bottom circulation in Buzzards Bay and consequent transport of fine-grained sediments that may be contaminated with PCBs from inner New Bedford Harbor.\n",
      "CAMP,B. Butman,California Area Monitoring Program (CAMP),A four-year multi-disciplinary field and laboratory study to investigate the sediment transport regime in the vicinity of production drilling rigs in the Santa Barbara Basin\n",
      "CAPE_COD_BAY,B. Butman,Currents and Sediment Transport in Cape Cod Bay,A pilot study to determine the effect of winter storms on sediment movement at two potential dredge spoil disposal areas.\n",
      "CC_MISC,B. Butman,Transport studies - Nauset Inlet,Part of a collaborative study of sediment movement in Nauset Inlet.\n",
      "DEEP_REEF,J. Lacey,Gulf of Mexico - Pinnacles,Pressure data from the Gulf of Mexico\n",
      "DWDS_106,B. Butman,Sediment Transport at Deep Water Dump Site 106,Near-bottom current measurements to understand the fate and transport of sludge from the New York Metropolitan region discharged at the sea surface.\n",
      "ECOHAB_II,R. Signell,Ecology of Harmful Algal Blooms (ECOHAB-II),A field program to continue investigating the transport and fate of toxic dinoflagellate blooms in the western Gulf of Maine.\n",
      "ECOHAB_I,R. Signell,Ecology of Harmful Algal Blooms (ECOHAB-I),A field program to study the transport and fate of toxic dinoflagellate blooms in the western Gulf of Maine.\n",
      "EUROSTRATAFORM,C. Sherwood,EuroSTRATAFORM,The EuroSTRATAFORM Po and Apennine Sediment Transport and Accumulation (PASTA) experiment was an international study of sediment-transport processes and formation of geological strata in the Adriatic Sea.\n",
      "FARALLONES,M. Noble,Farallons,Program to measure the currents and circulation on the continental slope off San Francisco CA and thus infer the transport of dredged materialat the newly-established deep-water disposal site.\n",
      "GB_SED,B. Butman,Georges Bank Current and Sediment Transport Studies,A series of studies to assess environmental hazards to petroleum development in the Georges Bank and New England Shelf region\n",
      "GLOBEC_GB,R. Schlitz,GLOBEC Georges Bank Program,A moored array program to investigate the circulation and mixing of plankton on Georges Bank.\n",
      "GLOBEC_GSC,R. Schlitz,GLOBEC Great South Channel Circulation Experiment,A moored array program to investigate the recirculation of water and plankton around Georges Bank\n",
      "GULF_MAINE,B. Butman,Deep Circulation in the Gulf of Maine,A two-year field study to investigate the deep flow between the major basins in the Gulf of Maine and the effects on the distribution of suspended sediments.\n",
      "HUDSON_SVALLEY,B. Butman,Circulation and Sediment Transport in the Hudson Shelf Valley,Field experiments have been carried out to understand the transport of sediments and associated contaminants in the Hudson Shelf Valley offshore of New York.\n",
      "KARIN_RIDGE,M. Noble,Karin Ridge Experiment,Current measurements collected at 2 sites in Karin Ridge Seamount.\n",
      "LYDONIA_C,B. Butman,Lydonia Canyon Dynamics Experiment,A major field experiment to determine the importance of submarine canyons in sediment transport along and across the continental margin.\n",
      "MAB_SED,B. Butman,Sediment Transport Observations in the Middle Atlantic Bight,A series of studies to assess environmental hazards to petroleum development in the Middle Atlantic Bight. \n",
      "MAMALA_BAY,D. Cacchione,Mamala bay Experiment,Current measurements collected at 350-450 meters in Mamala Bay near Waikiki Beach. \n",
      "MBAY_CIRC,R. Signell, Massachusetts Bay Circulation Experiment,Current measurements collected at 6 sites in Massachusetts Bay throughout the year to map the tidal wind and density driven currents. \n",
      "MBAY_IWAVE,B. Butman,Massachusetts Bay Internal Wave Experiment,A 1-month 4-element moored array experiment to measure the currents associated with large-amplitude internal waves generated by tidal flow across Stellwagen Bank.\n",
      "MBAY_LTB,B. Butman,Long-term observations in Massachusetts Bay; Site B-Scituate,Measurements of currents and other oceanographic properties were made to assess the impact of sewage discharge from the proposed outfall site.\n",
      "MBAY_LT,B. Butman,Long-term observations in Massachusetts Bay; Site A-Boston Harbor,Measurements of currents and other oceanographic properties were made to assess the impact of sewage discharge from the proposed outfall site.\n",
      "MBAY_STELL,R. Signell,Monitoring on Stellwagen Bank,A year-long series of current measurements on the eastern flank of Stellwagen Bank to document the currents at the mouth of Massachusetts Bay driven by the Maine Coastal current.\n",
      "MBAY_WEST,B. Butman,Currents and Sediment Transport in Western Massachusetts Bay,A pilot winter-time experiment to investigate circulation and sediment transport. Designed to provide information to aid in citing the new ocean outfall for the Boston sewer system.\n",
      "MOBILE_BAY,B. Butman,Mobile Bay Study,Measure currents and transport out of Mobile Bay.\n",
      "MONTEREY_BAY,M. Noble,Monterey Bay National Marine Sanctuary Program,Part of a large multi-disciplinary experiment to characterize the geologic environment and to generate a sediment budget.\n",
      "MONTEREY_CAN,M. Noble,Monterey Canyon Experiment, A program to determine the mechanisms that govern the circulation within and the transport of sediment and water through Monterey Submarine Canyon.\n",
      "MYRTLEBEACH,J. Warner,Myrtle Beach Experiment SC,Measurements collected as part of a larger study to understand the physical processes that control the transport of sediments in Long Bay South Carolina. \n",
      "NE_SLOPE,B. Butman,Currents on the New England Continental Slope,A study designed to describe the currents and to investigate the transport of sediment from the shelf to the slope. \n",
      "OCEANOG_C,B. Butman,Oceanographer Canyon Dynamics Experiment,A field experiment to determine the importance of submarine canyons in sediment transport along and across the continental margin.\n",
      "ORANGE_COUNTY,M. Noble,Orange County Sanitation District Studies,Observations to monitor coastal ocean process that transport suspended material and associated comtaminants across the shelf\n",
      "PONCHARTRAIN,R. Signell,Lake Ponchartrain Project,A series of moored array studies to investigate the circulation and particle transport in Lake Pontchartrain.\n",
      "PV_SHELF04,M. Noble,Palos Verdes Shelf 2004,Additional observations to estimate the quantity and direction of sediment erosion and transport on the shelf near the White Point ocean outfalls.\n",
      "PV_SHELF07,M. Noble,Palos Verdes Shelf 2007,Follow-up observations to evaluate how often coastal ocean processes move the DDT contaminated sediments near the White Point ocean outfalls.\n",
      "PV_SHELF,M. Noble,Palos Verdes Shelf Study,Initial observations of currents and circulation near the White Point ocean outfalls determine how often coastal ocean processes move the DDT contaminated sediments in this region.\n",
      "SAB_SED,B. Butman,Sediment Transport Observations in the Southern Atlantic Bight,A series of studies to assess environmental hazards to petroleum development in the South Atlantic Bight.\n",
      "SOUTHERN_CAL,M. Noble,Southern California Project,A series of moorings were deployed to understand how coastal ocean processes that move sediments change with location on the shelf.\n",
      "STRESS,B. Butman,Sediment Transport on Shelves and Slopes (STRESS),Experiment on the California continental margin to investigate storm-driven sediment transport.\n",
      "WRIGHTSVILLE,R. Thieler,Wrightsville Beach Study, Measurements of bottom currents and waves to investigate the flow field and sediment transport in a rippled scour depression offshore of Wrightsville Beach NC.\n",
      "DIAMONDSHOALS,J. Warner,Cape Hatteras- Diamond Shoals,This experiment was designed to investigate the ocean circulation and sediment transport dynamics at Diamond Shoals NC.\n",
      "CHANDELEUR,C. Sherwood,Chandeleur Islands Oceanographic Measurements,A program to measure waves water levels and currents near the Chandeleur Islands Louisiana and adjacent berm construction site.\n",
      "WFAL,N. Ganju,West Falmouth Harbor Fluxes,Oceanographic and water-quality observations made at six locations in West Falmouth Harbor and Buzzards Bay.\n",
      "BW2011,N. Ganju, Blackwater 2011, Oceanographic and Water-Quality Measurements made at several sites in 2 watersheds in Blackwater National Wildlife Refuge.\n",
      "MVCO_11,C. Sherwood, OASIS MVCO 2011, Near-seabed Oceanographic Observations made as part of the 2011 OASIS Project at the MVCO.\n",
      "HURRIRENE_BB,B. Butman, Observations in Buzzards Bay during and after a Hurricane, Oceanographic data collected in Buzzards Bay MA during Hurricane Irene August 2011.'''"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 77
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "project = pd.read_csv(StringIO.StringIO(projs.strip()), \n",
      "                      sep=\",\\s*\",index_col='project_id',\n",
      "                      names=['project_id', 'project_pi', 'project_name','project_summary'])"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 78
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "project.ix['PV_SHELF']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 9,
       "text": [
        "project_pi                                                  M. Noble\n",
        "project_name                                Palos Verdes Shelf Study\n",
        "project_summary    Initial observations of currents and circulati...\n",
        "Name: PV_SHELF, dtype: object"
       ]
      }
     ],
     "prompt_number": 9
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "len(project)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 57,
       "text": [
        "47"
       ]
      }
     ],
     "prompt_number": 57
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for index,row in project.iterrows():\n",
      "    print index,row['project_pi']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "ARGO_MERCHANT B. Butman\n",
        "BUZZ_BAY B. Butman\n",
        "CAMP B. Butman\n",
        "CAPE_COD_BAY B. Butman\n",
        "CC_MISC B. Butman\n",
        "DEEP_REEF J. Lacey\n",
        "DWDS_106 B. Butman\n",
        "ECOHAB_II R. Signell\n",
        "ECOHAB_I R. Signell\n",
        "EUROSTRATAFORM C. Sherwood\n",
        "FARALLONES M. Noble\n",
        "GB_SED B. Butman\n",
        "GLOBEC_GB R. Schlitz\n",
        "GLOBEC_GSC R. Schlitz\n",
        "GULF_MAINE B. Butman\n",
        "HUDSON_SVALLEY B. Butman\n",
        "KARIN_RIDGE M. Noble\n",
        "LYDONIA_C B. Butman\n",
        "MAB_SED B. Butman\n",
        "MAMALA_BAY D. Cacchione\n",
        "MBAY_CIRC R. Signell\n",
        "MBAY_IWAVE B. Butman\n",
        "MBAY_LTB B. Butman\n",
        "MBAY_LT B. Butman\n",
        "MBAY_STELL R. Signell\n",
        "MBAY_WEST B. Butman\n",
        "MOBILE_BAY B. Butman\n",
        "MONTEREY_BAY M. Noble\n",
        "MONTEREY_CAN M. Noble\n",
        "MYRTLEBEACH J. Warner\n",
        "NE_SLOPE B. Butman\n",
        "OCEANOG_C B. Butman\n",
        "ORANGE_COUNTY M. Noble\n",
        "PONCHARTRAIN R. Signell\n",
        "PV_SHELF04 M. Noble\n",
        "PV_SHELF07 M. Noble\n",
        "PV_SHELF M. Noble\n",
        "SAB_SED B. Butman\n",
        "SOUTHERN_CAL M. Noble\n",
        "STRESS B. Butman\n",
        "WRIGHTSVILLE R. Thieler\n",
        "DIAMONDSHOALS J. Warner\n",
        "CHANDELEUR C. Sherwood\n",
        "WFAL N. Ganju\n",
        "BW2011 N. Ganju\n",
        "MVCO_11 C. Sherwood\n",
        "hurrIrene_bb B. Butman\n"
       ]
      }
     ],
     "prompt_number": 61
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "\"\"\"function nname=lookup_cf(long_name)\n",
      "% LOOKUP_CF  Get CF equivalent name for EPIC variable long_name\n",
      "% return the new name string or [] if there's no equivalent\n",
      "%\n",
      "\n",
      "if(strfind(lower(long_name),'temp'))\n",
      "    nname='sea_water_temperature';\n",
      "elseif (strfind(lower(long_name),'cond'))\n",
      "    nname='sea_water_electrical_conductivity';\n",
      "elseif (strfind(lower(long_name),'sal'))\n",
      "    nname='sea_water_salinity';\n",
      "elseif (strfind(lower(long_name),'sigma'))\n",
      "    nname='sea_water_sigma_theta';\n",
      "% also have to deal with the min, max std of vels for burst stats files\n",
      "elseif (strfind(lower(long_name),'east'))\n",
      "    nname='eastward_sea_water_velocity';\n",
      "elseif (strfind(lower(long_name),'north'))\n",
      "    nname='northward_sea_water_velocity';\n",
      "elseif (strfind(lower(long_name),'vertical'))\n",
      "    nname='upward_sea_water_velocity';\n",
      "elseif (strfind(lower(long_name),'pitch'))\n",
      "    nname='platform_pitch_angle';\n",
      "elseif (strfind(lower(long_name),'roll'))\n",
      "    nname='platform_roll_angle';\n",
      "elseif (strfind(lower(long_name),'head'))\n",
      "    nname='platform_orientation';\n",
      "elseif (strfind(lower(long_name),'pres'))\n",
      "    if ~isempty(strfind(lower(long_name),'dev')) || ~isempty(strfind(lower(long_name),'std'))\n",
      "        nname=[];\n",
      "    else\n",
      "       nname='sea_water_pressure';\n",
      "    end\n",
      "elseif (strfind(lower(long_name),'cond'))\n",
      "    nname='sea_water_electrical_conductivity';\n",
      "elseif (strfind(lower(long_name),'speed'))\n",
      "    if (strfind(lower(long_name),'rotor'))\n",
      "        nname=[];\n",
      "    else\n",
      "       nname='sea_water_speed';\n",
      "    end\n",
      "elseif (strfind(lower(long_name),'direction'))\n",
      "    nname='direction_of_sea_water_velocity';\n",
      "else\n",
      "    nname=[];\n",
      "end\n",
      "disp([long_name ' : ' nname])\n",
      "\"\"\";"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 10
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "#os.chdir('/usgs/data2/emontgomery/stellwagen/Data/ARGO_MERCHANT')\n",
      "\n",
      "root_dir='/usgs/data2/emontgomery/stellwagen/Data/'\n",
      "#root_dir='/usgs/data2/emontgomery/stellwagen/Data/MVCO_11'\n",
      "odir='/usgs/data2/emontgomery/stellwagen/CF-1.6/'\n",
      "os.chdir(root_dir)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 40
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for path, subdirs, files in os.walk(root_dir,topdown=True):\n",
      "    subdirs[:]=[]\n",
      "    print subdirs\n",
      "    for name in files:\n",
      "        file= os.path.join(path, name)\n",
      "        print file"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[]\n",
        "/usgs/data2/emontgomery/stellwagen/Data/.DS_Store\n"
       ]
      }
     ],
     "prompt_number": 13
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# now find all the unique names, long_names & units\n",
      "names = set()\n",
      "long_names = set()\n",
      "units = set()\n",
      "for path, subdirs, files in os.walk(root_dir):\n",
      "    for name in files:\n",
      "        file= os.path.join(path, name)\n",
      "        try:\n",
      "            nc=netCDF4.Dataset(file)\n",
      "            for var in nc.variables.keys():\n",
      "                names.add(var)\n",
      "                try:\n",
      "                    long_names.add(nc.variables[var].long_name)\n",
      "                except:\n",
      "                    pass\n",
      "                try:\n",
      "                    units.add(nc.variables[var].units)\n",
      "                except:\n",
      "                    pass\n",
      "                    \n",
      "        except:\n",
      "            pass\n",
      "\n",
      "names= list(names)\n",
      "long_names = list(long_names)\n",
      "units = list(units)\n",
      "\n",
      "print len(names)\n",
      "print len(long_names)\n",
      "print len(units)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "509\n",
        "545\n",
        "153\n"
       ]
      }
     ],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 14
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# let's use Ellyn's approach of matching substrings in the long_names to deduce standard_names"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 15
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# standard_name to variable mapping\n",
      "d={}\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 16
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# air temp, frtemp, laser temp factor, internal, temp diff\n",
      "filter(lambda x:re.search(r'temp',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 17,
       "text": [
        "[u'AIR TEMPERATURE (C)      ',\n",
        " u'instrument Transducer Temp.',\n",
        " u'Air Temperature (degrees C)',\n",
        " u'TEMPERATURE (C)          ',\n",
        " u'FR TEMP     ',\n",
        " u'ADP Transducer Temp.',\n",
        " u'ADCP Transducer Temp.',\n",
        " u'TEMPERATURE (C)',\n",
        " u'Transducer Temp.    ',\n",
        " u'laser temperature factor',\n",
        " u'Transducer Temp.',\n",
        " u'TEMP 1      ',\n",
        " u'TEMP 2      ',\n",
        " u'Temperature',\n",
        " u'internal temperature',\n",
        " u'FRTEMP      ',\n",
        " u'TEMP 2 Q    ',\n",
        " u'TEMP        ',\n",
        " u'TEMP DIFF   ',\n",
        " u'TEMP LP     ',\n",
        " u'Sea Surface Temperature (degrees C)',\n",
        " u'TEMPERATURE ']"
       ]
      }
     ],
     "prompt_number": 17
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "d['sea_water_temperature']=['instrument transducer temp.', 'temperature (c)','fr temp',\n",
      "                            'adp transducer temp.','adcp transducer temp.','transducer temp.','temp 1','temp 2',\n",
      "                            'temperature','internal temperature','frtemp','temp 2 q','temp','temp lp','sea surface temperature (degrees C)']\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 18
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# seconds, second\n",
      "filter(lambda x:re.search(r'cond',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 19,
       "text": [
        "[u'SPECIFIC CONDUCTANCE        ',\n",
        " u'CONDUCTIVITY             ',\n",
        " u'Specific conductance',\n",
        " u'SECONDS SINCE START',\n",
        " u'second',\n",
        " u'CONDUCTIVITY',\n",
        " u'SP COND     ',\n",
        " u'COND        ',\n",
        " u'CTD Conductivity       ',\n",
        " u'COND 2      ',\n",
        " u'COND 1      ']"
       ]
      }
     ],
     "prompt_number": 19
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filter(lambda x:re.search(r'sal',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 20,
       "text": [
        "[u'SALINITY 2 Q',\n",
        " u'SALINITY 1  ',\n",
        " u'CTD Salinity, PSS-78',\n",
        " u'Salinity',\n",
        " u'SALINITY 2  ',\n",
        " u'SALINITY (PPT)           ',\n",
        " u'SALINITY (PSU)           ',\n",
        " u'SALINITY    ']"
       ]
      }
     ],
     "prompt_number": 20
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "d['sea_water_salinity'] = ['salinity 2 q','salinity 1','ctd salinity, pss-78','salinity','salinity (ppt)','salinity (psu)','salinity']\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 21
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filter(lambda x:re.search(r'sigma',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 22,
       "text": [
        "[u'SIGMA-THETA (KG/M**3) ',\n",
        " u'SIGMA-T (KG/M**3)        ',\n",
        " u'SIGMA-THETA (KG/M**3)    ',\n",
        " u'SIGMA THETA ']"
       ]
      }
     ],
     "prompt_number": 22
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# deal with burst data, std dev, resolution velocity, variance\n",
      "filter(lambda x:re.search(r'east',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 23,
       "text": [
        "[u'Eastward Velocity       ',\n",
        " u'Eastward Velocity',\n",
        " u'Ratio of variances for burst after points replaced by delgitch: Eastward Velocity',\n",
        " u'EAST        ',\n",
        " u'Maximum Eastward Velocity',\n",
        " u'Minimum Eastward Velocity',\n",
        " u'Eastward Resolution Velocity',\n",
        " u'Std. Dev. of east component',\n",
        " u'Ratio of means for burst after points replaced by delgitch: Eastward Velocity',\n",
        " u'EAST VELOCITY VARIANCE   ',\n",
        " u'Mean Eastward Velocity',\n",
        " u'Eastward Velocity        ',\n",
        " u'Eastward Velocity         ',\n",
        " u'EAST LP     ',\n",
        " u'EAST VELOCITY VARIANCE ',\n",
        " u'Mean Resolution VelocityEastward']"
       ]
      }
     ],
     "prompt_number": 23
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "d['eastward_sea_water_velocity']=['eastward velocity','east','mean eastward velocity','eastward velocity','east lp']\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 24
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filter(lambda x:re.search(r'north',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 25,
       "text": [
        "[u'NORTH       ',\n",
        " u'Northward Velocity        ',\n",
        " u'Std. Dev. of north component',\n",
        " u'Peak Wave Direction (degrees North)',\n",
        " u'Wind Direction (degrees true North)',\n",
        " u'Ratio of means for burst after points replaced by delgitch: Northward Velocity',\n",
        " u'Minimum Northward Velocity',\n",
        " u'Maximum Northward Velocity',\n",
        " u'Peak Wave Direction (degrees true North)',\n",
        " u'Northward Velocity',\n",
        " u'Mean Wave Direction (degrees true North)',\n",
        " u'Mean Resolution VelocityNorthward',\n",
        " u'NORTH VELOCITY VARIANCE ',\n",
        " u'NORTH LP    ',\n",
        " u'Northward Resolution Velocity',\n",
        " u'Mean Northward Velocity',\n",
        " u'Ratio of variances for burst after points replaced by delgitch: Northward Velocity',\n",
        " u'Northward Velocity      ',\n",
        " u'Northward Velocity       ',\n",
        " u'NORTH VELOCITY VARIANCE  ']"
       ]
      }
     ],
     "prompt_number": 25
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "d['northward_sea_water_velocity']=['northward velocity','north','mean northward velocity','northward velocity','north lp']\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 26
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filter(lambda x:re.search(r'vertical',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 27,
       "text": [
        "[u'Vertical Velocity    ',\n",
        " u'Vertical Velocity        ',\n",
        " u'Ratio of means for burst after points replaced by delgitch: Vertical Velocity',\n",
        " u'Vertical Resolution Velocity',\n",
        " u'Ratio of variances for burst after points replaced by delgitch: Vertical Velocity',\n",
        " u'Vertical Velocity',\n",
        " u'Maximum Vertical Velocity',\n",
        " u'Mean Resolution VelocityVertical',\n",
        " u'Vertical Velocity         ',\n",
        " u'Mean Vertical Velocity',\n",
        " u'Minimum Vertical Velocity']"
       ]
      }
     ],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# wind, rotor speed\n",
      "filter(lambda x:re.search(r'speed',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 28,
       "text": [
        "[u'WIND SPEED (CM/S)        ',\n",
        " u'ROTOR SPEED DIFFERENCE',\n",
        " u'ROTOR SPEED DIFFERENCE  ',\n",
        " u'Current Speed',\n",
        " u'UPPER ROTOR SPEED     ',\n",
        " u'Wind Speed (m/s)',\n",
        " u'CURRENT SPEED (CM/S)      ',\n",
        " u'LOWER ROTOR SPEED  ',\n",
        " u'CURRENT SPEED (CM/S)    ',\n",
        " u'VECTOR SPEED',\n",
        " u'LOWER ROTOR SPEED    ',\n",
        " u'WIND SPEED (M/S) ',\n",
        " u'WIND SPEED (CM/S) ',\n",
        " u'CURRENT SPEED (CM/S)       ',\n",
        " u'ROTOR SPEED             ',\n",
        " u'ROTOR SPEED ']"
       ]
      }
     ],
     "prompt_number": 28
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# std dev\n",
      "filter(lambda x:re.search(r'pitch',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 29,
       "text": [
        "[u'INST Pitch',\n",
        " u'Pitch Std. Dev.          ',\n",
        " u'Standard Deviation of INST Pitch',\n",
        " u'INST Pitch             ']"
       ]
      }
     ],
     "prompt_number": 29
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filter(lambda x:re.search(r'roll',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 30,
       "text": [
        "[u'Roll Std. Dev.             ',\n",
        " u'INST Roll             ',\n",
        " u'INST Roll',\n",
        " u'Standard Deviation of INST Roll']"
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# std dev\n",
      "filter(lambda x:re.search(r'heading',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 31,
       "text": [
        "[u'Median INST Heading             ',\n",
        " u'Heading Std. Dev.        ',\n",
        " u'INST Heading',\n",
        " u'INST Heading             ']"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# std, dev, wave height spectra, barometric, presscheck\n",
      "pres = filter(lambda x:re.search(r'press',x.lower()), long_names)\n",
      "pres"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 32,
       "text": [
        "[u'STAND.DEV.(PRESS)',\n",
        " u'Pressure at Transducer Head',\n",
        " u'Pressure',\n",
        " u'PRESSURE (PASCALS)',\n",
        " u'PRESSURE (DB)          ',\n",
        " u'STAND. DEV. (PRESS)     ',\n",
        " u'AVERAGE BURST PRESSURE   ',\n",
        " u'Bottom Pressure (psia)           ',\n",
        " u'BAROMETRIC PRESSURE (MB) ',\n",
        " u'AVERAGE BURST PRESSURE  ',\n",
        " u'Pressure-derived Non-directional Wave Height Spectrum (mm/sqrt(Hz))',\n",
        " u'STAND. DEV. (PRESS)    ',\n",
        " u'INTERVAL PRESSURE       ',\n",
        " u'Barometric Pressure (mbar)',\n",
        " u'PRESS       ',\n",
        " u'Barometric Pressure (mb)',\n",
        " u'ADCP Pressure at Transducer Head',\n",
        " u'AVERAGE RELATIVE PRESSURE',\n",
        " u'PRESSURE            ',\n",
        " u'Raw pressure Std. Dev.',\n",
        " u'PRESSCHECK  ',\n",
        " u'INTERVAL PRESSURE        ',\n",
        " u'PRESSURE    ']"
       ]
      }
     ],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "filter(lambda x:re.search(r'std|dev',x.lower()), pres)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 33,
       "text": [
        "[u'STAND.DEV.(PRESS)',\n",
        " u'STAND. DEV. (PRESS)     ',\n",
        " u'STAND. DEV. (PRESS)    ',\n",
        " u'Raw pressure Std. Dev.']"
       ]
      }
     ],
     "prompt_number": 33
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# save only direction and current direction\n",
      "filter(lambda x:re.search(r'direct',x.lower()), long_names)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 34,
       "text": [
        "[u'beginning of first direction slice, degrees',\n",
        " u'Surface-derived Non-directional Wave Height Spectrum (mm/sqrt(Hz))',\n",
        " u'Direction (degrees)',\n",
        " u'Peak Wave Direction (degrees North)',\n",
        " u'Wind Direction (degrees true North)',\n",
        " u'V  DIRECTION',\n",
        " u'Peak Wave Direction (degrees true North)',\n",
        " u'CURRENT DIRECTION (T)      ',\n",
        " u'WIND DIRECTION           ',\n",
        " u'CURRENT DIRECTION (T)     ',\n",
        " u'Pressure-derived Non-directional Wave Height Spectrum (mm/sqrt(Hz))',\n",
        " u'WIND DIRECTION STABILITY',\n",
        " u'Mean Wave Direction (degrees true North)',\n",
        " u'Directional Wave Energy Spectrum (mm^2/Hz/degree)',\n",
        " u'Velocity-derived Non-directional Wave Height Spectrum (mm/sqrt(Hz))',\n",
        " u'mean wave direction',\n",
        " u'Direction (degrees T)',\n",
        " u'WIND DIRECTION        ',\n",
        " u'Current Direction',\n",
        " u'CURRENT DIRECTION (T)   ']"
       ]
      }
     ],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "f = open('/usgs/data2/notebook/names.txt','w')\n",
      "f.write(\"\\n\".join(names))\n",
      "f.close()\n",
      "f = open('/usgs/data2/notebook/long_names.txt','w')\n",
      "f.write(\"\\n\".join(long_names))\n",
      "f.close()\n",
      "f = open('/usgs/data2/notebook/units.txt','w')\n",
      "f.write(\"\\n\".join(units))\n",
      "f.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 35
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 35
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "pwd"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 36,
       "text": [
        "u'/usgs/data2/emontgomery/stellwagen/Data'"
       ]
      }
     ],
     "prompt_number": 36
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "root_dir='/usgs/data2/emontgomery/stellwagen/Data/'\n",
      "#root_dir='/usgs/data2/emontgomery/stellwagen/Data/MVCO_11'\n",
      "odir='/usgs/data2/emontgomery/stellwagen/CF-1.6/'\n",
      "os.chdir(root_dir)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 66
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "types = ('*.nc', '*.cdf')\n",
      "for index,row in project.iterrows():\n",
      "    os.chdir(os.path.join(root_dir,index))\n",
      "    ncfiles=[]\n",
      "    for files in types:\n",
      "        ncfiles.extend(glob.glob(files))\n",
      "    print index, len(ncfiles)\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "ARGO_MERCHANT 6\n",
        "BUZZ_BAY 81\n",
        "CAMP 42\n",
        "CAPE_COD_BAY 34\n",
        "CC_MISC 2\n",
        "DEEP_REEF 1\n",
        "DWDS_106 16\n",
        "ECOHAB_II 111\n",
        "ECOHAB_I 86\n",
        "EUROSTRATAFORM 4\n",
        "FARALLONES 41\n",
        "GB_SED 242\n",
        "GLOBEC_GB 37\n",
        "GLOBEC_GSC 42\n",
        "GULF_MAINE 35\n",
        "HUDSON_SVALLEY 167\n",
        "KARIN_RIDGE 14\n",
        "LYDONIA_C 230\n",
        "MAB_SED 130\n",
        "MAMALA_BAY 33\n",
        "MBAY_CIRC 50\n",
        "MBAY_IWAVE 31\n",
        "MBAY_LTB 85\n",
        "MBAY_LT 1416\n",
        "MBAY_STELL 36\n",
        "MBAY_WEST 29\n",
        "MOBILE_BAY 98\n",
        "MONTEREY_BAY 122\n",
        "MONTEREY_CAN 33\n",
        "MYRTLEBEACH 79\n",
        "NE_SLOPE 117\n",
        "OCEANOG_C 14\n",
        "ORANGE_COUNTY 55\n",
        "PONCHARTRAIN 35\n",
        "PV_SHELF04 58\n",
        "PV_SHELF07 71\n",
        "PV_SHELF 87\n",
        "SAB_SED 32\n",
        "SOUTHERN_CAL 30\n",
        "STRESS 108\n",
        "WRIGHTSVILLE 13\n",
        "DIAMONDSHOALS 39\n",
        "CHANDELEUR 16\n",
        "WFAL 28\n",
        "BW2011 12\n",
        "MVCO_11 13\n",
        "HURRIRENE_BB 4\n"
       ]
      }
     ],
     "prompt_number": 79
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "##Read a sample old file "
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "file = files[-1]\n",
      "print file\n",
      "nc = netCDF4.Dataset(file)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "9111aqd-a.nc\n"
       ]
      }
     ],
     "prompt_number": 42
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "vars=nc.variables.keys()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 43
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "coord_vars = ['time','time2','depth','lat','lon']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 44
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# find data variables by removing coordinate variables from the variable list\n",
      "data_vars = [var for var in vars if var not in coord_vars]\n",
      "print data_vars"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "[u'bindist', u'u_1205', u'v_1206', u'w_1204', u'CS_300', u'CD_310', u'P_1', u'Tx_1211', u'AGC_1202', u'Hdg_1215', u'Ptch_1216', u'Roll_1217']\n"
       ]
      }
     ],
     "prompt_number": 45
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "print nc.variables['u_1205']"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "<type 'netCDF4.Variable'>\n",
        "float32 u_1205(time, depth, lat, lon)\n",
        "    name: u\n",
        "    long_name: Eastward Velocity\n",
        "    feneric_name: u\n",
        "    FORTRAN_format:  \n",
        "    units: cm/s\n",
        "    epic_code: 1205\n",
        "    sensor_type: Nortek Aquadopp Profiler\n",
        "    sensor_depth: 12.8568007307\n",
        "    initial_sensor_height: 0.16\n",
        "    height_depth_units: m\n",
        "    serial_number: AQD 6409\n",
        "    minimum: [ -37.79090881  -39.59309769  -39.48016739  -40.98865509  -41.56013489\n",
        "  -42.0097084   -42.66938019  -42.61388397  -43.71333313  -41.31264114\n",
        "  -46.47284317  -44.69533157  -47.77171707  -49.01902008  -52.09646988\n",
        "  -54.16630173  -54.65877533  -58.35764313  -58.2725029   -57.30296707\n",
        "  -72.47348022 -102.24491882 -108.71712494 -107.08737183  -92.03878784]\n",
        "    maximum: [ 34.93043518  31.40140343  34.67355347  38.76890182  39.16253662\n",
        "  42.82086945  45.19152069  42.90264511  47.63571548  47.8465271\n",
        "  49.28278732  45.30502319  49.03020096  47.69023514  47.4201355\n",
        "  45.45179367  45.16744232  44.54811096  45.65585709  47.87711334\n",
        "  44.75069046  48.67462158  71.71102905  78.26560974  69.04908752]\n",
        "    valid_range: []\n",
        "    _FillValue: 1e+35\n",
        "unlimited dimensions: time\n",
        "current shape = (5214, 25, 1, 1)\n",
        "\n"
       ]
      }
     ],
     "prompt_number": 46
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "nt = len(nc.dimensions['time'])\n",
      "nz = len(nc.dimensions['depth'])\n",
      "print nt,nz"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "5214 25\n"
       ]
      }
     ],
     "prompt_number": 47
    },
    {
     "cell_type": "markdown",
     "metadata": {},
     "source": [
      "## Write a sample \"CF-1.6, timeSeries\" File"
     ]
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ofile = '/usgs/data2/notebook/%s' % file\n",
      "id = file.split('.')[0]\n",
      "print id"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "9111aqd-a\n"
       ]
      }
     ],
     "prompt_number": 48
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# create dimensions\n",
      "nco = netCDF4.Dataset(ofile,'w',clobber=True)\n",
      "nco.createDimension('time',nt)\n",
      "if nz > 1:\n",
      "    nco.createDimension('depth',nz)\n",
      "\n",
      "nchar=20\n",
      "nco.createDimension('nchar',nchar)"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 49,
       "text": [
        "<netCDF4.Dimension at 0x3b1ac80>"
       ]
      }
     ],
     "prompt_number": 49
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# create coordinate variables\n",
      "time_v = nco.createVariable('time', 'f8', ('time'))\n",
      "lon_v = nco.createVariable('lon','f4')\n",
      "lat_v = nco.createVariable('lat','f4')\n",
      "if nz > 1:\n",
      "    depth_v = nco.createVariable('depth','f4',dimensions='depth')\n",
      "else:\n",
      "    depth_v = nco.createVariable('depth','f4')\n",
      "\n",
      "station_v = nco.createVariable('site','S1',('nchar'))"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 50
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# write global attributes\n",
      "g_attdict = nc.__dict__\n",
      "g_attdict['Conventions'] = 'CF-1.6'\n",
      "if nz>1:\n",
      "    g_attdict['featureType'] = 'timeSeriesProfile'\n",
      "else:\n",
      "    g_attdict['featureType'] = 'timeSeries'\n",
      "\n",
      "g_attdict['naming_authority'] = 'gov.usgs'\n",
      "g_attdict['id'] = id\n",
      "g_attdict['source'] = 'USGS'\n",
      "g_attdict['institution'] = 'Woods Hole Coastal and Marine Science Center'\n",
      "g_attdict['project'] = 'Coastal and Marine Geology Program'\n",
      "g_attdict['title'] = g_attdict['source'] + g_attdict['id']\n",
      "g_attdict['keywords']='Oceans > Ocean Pressure > Water Pressure, Oceans > Ocean Temperature > Water Temperature, Oceans > Salinity/Density > Conductivity, Oceans > Salinity/Density > Salinity'\n",
      "g_attdict['keywords_vocabulary']='GCMD Science Keywords'\n",
      "g_attdict['standard_name_vocabulary'] = 'CF-1.6'\n",
      "g_attdict['creator_email'] = 'emontgomery@usgs.gov'\n",
      "g_attdict['creator_name'] =  'Ellyn Montgomery'\n",
      "\n",
      "nco.setncatts(g_attdict) "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 51
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# write station variable\n",
      "station_v.cf_role = 'timeseries_id'\n",
      "station_v.standard_name = 'station_id'\n",
      "data = numpy.empty((1,),'S'+repr(nchar))\n",
      "data[0] = id\n",
      "station_v[:] = netCDF4.stringtochar(data)\n",
      "\n",
      "# write time variable\n",
      "time_v.units = 'seconds since 1858-11-17 00:00:00 +0:00'\n",
      "time_v.standard_name = 'time'\n",
      "time_v.calendar = 'gregorian'\n",
      "time_v[:] = (nc.variables['time'][:]-2400001.)*3600.*24. + nc.variables['time2'][:]/1000.\n",
      "\n",
      "# write lon variable\n",
      "lon_v.units = 'degree_east'\n",
      "lon_v.standard_name = 'longitude'\n",
      "lon_v[:] = nc.variables['lon'][:]\n",
      "\n",
      "# write lat variable\n",
      "lat_v.units = 'degree_north'\n",
      "lat_v.standard_name = 'latitude'\n",
      "lat_v[:] = nc.variables['lat'][:]\n",
      "\n",
      "# write depth variable\n",
      "depth_v.units = 'm'\n",
      "depth_v.standard_name = 'depth'   \n",
      "depth_v.positive = 'down'\n",
      "depth_v.axis = 'Z'\n",
      "depth_v[:] = nc.variables['depth'][:]\n",
      "\n"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 52
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "nc.variables['depth'][:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 53,
       "text": [
        "array([ 12.35680073,  11.85680073,  11.35680073,  10.85680073,\n",
        "        10.35680073,   9.85680073,   9.35680073,   8.85680073,\n",
        "         8.35680073,   7.85680073,   7.35680073,   6.85680073,\n",
        "         6.35680073,   5.85680073,   5.35680073,   4.85680073,\n",
        "         4.35680073,   3.85680073,   3.35680073,   2.85680073,\n",
        "         2.35680073,   1.85680073,   1.35680073,   0.85680073,   0.35680073])"
       ]
      }
     ],
     "prompt_number": 53
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "# create the data variables\n",
      "var_v=[]\n",
      "for varname in data_vars:\n",
      "    ncvar = nc.variables[varname]\n",
      "    # if time series variable\n",
      "    if size(ncvar) == nt:\n",
      "        var = nco.createVariable(varname,ncvar.dtype,('time'))\n",
      "    elif size(ncvar) == nz:\n",
      "        var = nco.createVariable(varname,ncvar.dtype,('depth'))\n",
      "    else:\n",
      "        var = nco.createVariable(varname,ncvar.dtype,('time','depth'))\n",
      "\n",
      "#   load old variable attributes and modify if necessary \n",
      "    attdict = ncvar.__dict__\n",
      "#   if dounpackshort and 'scale_factor' in attdict: del attdict['scale_factor']\n",
      "  \n",
      "    attdict['coordinates'] = 'time lon lat depth'\n",
      "\n",
      "    # assign standard_name if in dictionary\n",
      "    a =[k for (k, v) in d.iteritems() if attdict['long_name'].strip().lower() in v]\n",
      "    if len(a)==1: attdict['standard_name']=a[0]\n",
      "    \n",
      "    # write variable attributes\n",
      "    var.setncatts(attdict) \n",
      "    # write the data\n",
      "#    print ncvar\n",
      "    var[:] = ncvar[:]"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 54
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "nco.close()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 55
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "root_dir='/usgs/data2/emontgomery/stellwagen/Data/MVCO_11'"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 27
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "cd /usgs/data2/emontgomery/stellwagen/Data/MVCO_11"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "/usgs/data2/emontgomery/stellwagen/Data/MVCO_11\n"
       ]
      }
     ],
     "prompt_number": 30
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "ls"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "9102sc-a.nc         9104pcs-cal.cdf   9106advs-cal.cdf  9111aqd-a.nc\r\n",
        "9103HRaqdb-cal.cdf  9105advb-cal.cdf  9107sc-a.nc\r\n",
        "9103HRaqds-cal.cdf  9105advs-cal.cdf  \u001b[0m\u001b[01;32m9107sc-tst2.nc\u001b[0m*\r\n",
        "9104pcb-cal.cdf     9106advb-cal.cdf  \u001b[01;32m9107sc-tst.nc\u001b[0m*\r\n"
       ]
      }
     ],
     "prompt_number": 31
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "nc = netCDF4.Dataset('9104pcs-cal.cdf')"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 32
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "for (k,v) in nc.variables.iteritems():\n",
      "    try: \n",
      "        print v.long_name\n",
      "    except:\n",
      "        pass\n",
      "    "
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "output_type": "stream",
       "stream": "stdout",
       "text": [
        "Burst Number\n",
        "EPIC SYSTEM TIME\n",
        "msec since 0:00 GMT\n",
        "DEPTH (m)\n",
        "LONGITUDE\n",
        "LATITUDE\n",
        "Mean Eastward Velocity\n",
        "Mean Northward Velocity\n",
        "Mean Vertical Velocity\n",
        "Std. Dev. of east component\n",
        "Std. Dev. of north component\n",
        "Std. Dev. of upward component\n",
        "Minimum Eastward Velocity\n",
        "Minimum Northward Velocity\n",
        "Minimum Vertical Velocity\n",
        "Maximum Eastward Velocity\n",
        "Maximum Northward Velocity\n",
        "Maximum Vertical Velocity\n",
        "CURRENT SPEED (CM/S)      \n",
        "CURRENT DIRECTION (T)     \n",
        "Mean Resolution VelocityEastward\n",
        "Mean Resolution VelocityNorthward\n",
        "Mean Resolution VelocityVertical\n",
        "Echo Intensity (AGC) Beam 1\n",
        "Echo Intensity (AGC) Beam 2\n",
        "Echo Intensity (AGC) Beam 3\n",
        " Beam 1 Correlation\n",
        " Beam 2 Correlation\n",
        " Beam 3 Correlation\n",
        "Sensor Range to Boundary\n",
        "Median INST Heading             \n",
        "INST Pitch             \n",
        "INST Roll             \n",
        "Heading Std. Dev.        \n",
        "Pitch Std. Dev.          \n",
        "Roll Std. Dev.             \n",
        "Transducer Temp.\n",
        "Distance from transducer head to bin center\n",
        "ATTENUATION 1\n",
        "TRANSMISSION (VOLTS) #1\n",
        "raw Backscattering 2\n",
        "BACKSCATTER INTENSITY 2\n"
       ]
      }
     ],
     "prompt_number": 34
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a='FOOBAR   '"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [],
     "prompt_number": 36
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [
      "a.strip().lower()"
     ],
     "language": "python",
     "metadata": {},
     "outputs": [
      {
       "metadata": {},
       "output_type": "pyout",
       "prompt_number": 37,
       "text": [
        "'foobar'"
       ]
      }
     ],
     "prompt_number": 37
    },
    {
     "cell_type": "code",
     "collapsed": false,
     "input": [],
     "language": "python",
     "metadata": {},
     "outputs": []
    }
   ],
   "metadata": {}
  }
 ]
}