Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

data not tagged with lat , long #170

Closed
jaigsingla opened this issue Sep 25, 2017 · 27 comments
Closed

data not tagged with lat , long #170

jaigsingla opened this issue Sep 25, 2017 · 27 comments
Assignees
Milestone

Comments

@jaigsingla
Copy link

@jaigsingla jaigsingla commented Sep 25, 2017

Hello All,

I have different variable for lat, long and other parameter.

Suppose I have a grid for lat (380,360) same for long and same for temp (380,360,14,1)

How to make it convenient for displaying it using surface plots.

I am able to read all variables and reshape these.
How to pack these as a matrix for plotting??

thanks
Jai

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Sep 25, 2017

@dnadeau4 let's make a notebook to explain this and post on website. @dnadeau4 if I remember correctly you did similar thing for Donald Luca earlier, do you still have the code?

@doutriaux1 doutriaux1 added this to the 3.0 milestone Sep 25, 2017
@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Sep 28, 2017

Hello
Any update related to this

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Sep 28, 2017

@jaigsingla I think @dnadeau4 is busy with over things at the moment, so I took on me to prepare a quick notebook to show how to do this, please let me know if that's clear and what you want.
Once you're happy with it I will clean it up and post it in our tutorials sections.

curv.tar.gz

raw code:

# coding: utf-8

# In[1]:


# Prepare the notebook elements
import requests
fnm = "sampleCurveGrid4.nc"
r = requests.get("https://uvcdat.llnl.gov/cdat/sample_data/%s" % fnm,stream=True)
with open(fnm,"wb") as f:
    for chunk in r.iter_content(chunk_size=1024):
        if chunk:  # filter local_filename keep-alive new chunks
            f.write(chunk)

# and load data
import cdms2
f = cdms2.open(fnm)
data = f("sample").filled() # Get data and forget all about it (pure numpy)
lat = f("lat").filled() # latitude coordinates
lon = f("lon").filled() # longitude coords
blon = f("bounds_lon").filled() # longitude vertices
blat = f("bounds_lat").filled() #latitude vertices


# In[2]:


# Let's prepare the "axes" for our data (i.e 2D lat lon)

lat_axis = cdms2.coord.TransientAxis2D(lat)
lat_axis.setBounds(blat)

lon_axis = cdms2.coord.TransientAxis2D(lon)
lon_axis.setBounds(blon)


# In[3]:


# Now let's create the grid
grid = cdms2.hgrid.TransientCurveGrid(lat_axis,lon_axis)


# In[4]:


# Finally let's prepare the data
data = cdms2.MV2.array(data)
data.setAxisList(grid.getAxisList())
data.setGrid(grid)


# In[5]:


# And llet's plot it to be sure it worked
import vcs
mesh = vcs.createmeshfill()
x=vcs.init(bg=True)
x.plot(data,mesh)

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Sep 29, 2017

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Sep 29, 2017

One more point data in my file is 4d whereas sample data used by you is 2d. Now I want to plot it in 3d surface

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Sep 29, 2017

@jaigsingla I can't help you more with vertices, you need to ask your data provider, if it's 3 or 4D it's the same concept simply set edit the part where you set the axis instead f setAxisList, you probably want to set axes one at a time with setAxis(index,axis)

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Sep 29, 2017

I'll update Jupyter notebook to show 4D data before posting it.

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Sep 29, 2017

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Sep 29, 2017

there's no way to guess especially on curvilinear and generic grids. You can try to use the mid point between two (i,j),(i+1,j+1) but that's probably going to be a poor guess. You have to know your grid.
On the bright side I think isofill/isoline shouldn't need vertices for plotting I will try to push a fix to vcs for such cases soon. @danlipsa am I guessing right that isofill/isoline shouldn't need vertices.

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 5, 2017

Hello @doutriaux1 ,
Actually I have latitude and longitude as separate fields and temp as separate field. Now, want to write in same data structure as used in geos5.nc file. Can not we achieve the same?

Latitude , longitude values can be extracted from corresponding array index and similarly temp profile (vertical 14 values) can be extracted from temp field and can be arranged in a data structure as that used in geos5.nc .

How do bounds field help in this regard? pls shed a light on this also .

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 5, 2017

bounds are for plotting. Can you please send a link to a geos5.nc file. I'm not 100% I follow what you're trying to achieve.

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 5, 2017

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 5, 2017

oh that! doh!

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 5, 2017

Yeah. Now in this file data is written well based on latitude longitude various values in different layers. But in my case data is not written like this , challenge to form this type of structure from my data file . I hope it's clear now

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 6, 2017

@doutriaux1 waiting for your comments
import cdms2,numpy as np

f=cdms2.open('/home/jai/miniconda2/sample_data/3D.h5')
#f.close()
out=np.zeros((320,384,42))
var=f['TAirPhy']
lat=f['Latitude']
lon=f['Longitude']

lat=np.reshape(lat,(320,384))
lon=np.reshape(lon,(320,384))

var=np.reshape(var,(320,384,40,1))

print var.shape, lat.shape, lon.shape

tmp=np.zeros((1,42))
t0=np.zeros(40)
for i in range(1,320):
for j in range(1,384):
#for k in range(1,40):
t0=np.float32(var[i][j][:])
#tmp=np.array([lat[i,j],lon[i,j],np.array(t0[:])])
tmp=np.array([lat[i,j],lon[i,j]])
tmp=np.append(tmp,[t0])
out[i,j,:]=np.float32([tmp])

out=np.reshape(out,(1,42,384,320))
g=cdms2.open('/home/jai/miniconda2/sample_data/tst.nc','w')
g.write(out)
g.close()

print lat[0][0],lon[0][0],out[0][0][:][0]

This code I am trying to restructure data . please comment

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 8, 2017

Hello @doutriaux1 ,

DO you need more information or input file ??

Awaiting reply .
Pls reply

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 12, 2017

Hello @doutriaux1 and @dnadeau4

pls find my input file for which I am trying to rearrange data.

sample.zip

pls help me out.

@jypeter
Copy link
Member

@jypeter jypeter commented Oct 12, 2017

@jaigsingla unfortunately I don't have much time to help you but here is something you can try

  1. read your data and associate it to a grid, following Charles example up to data.setGrid(grid). Don't use lat_axis.setBounds(blat) if you don't have bounds
  2. write this data to a netcdf file and check the content of the file with 'ncdump -h'
  3. in another script, read the data from the netcdf file and try to plot it with the 3d scalar examples available in the gallery: https://uvcdat.llnl.gov/gallery.html#3d_scalar. It's possible that the 3D methods don't need the bounds information

If you want more details about creating curvilinear grids and variables, you can go to http://www.ifremer.fr/vacumm/library/misc.grid.misc.html#vacumm.misc.grid.misc.create_axes2d and then click on [source]. Check what the other functions do as well

@doutriaux1 and @dnadeau4 there is a lot of good cdms2 stuff in what @stefraynaud has done and in the vacumm gallery. Maybe there should be links to (some parts of) vacumm from the uv-cdat site. I was able to install vacumm in uv-cdat 2.10 after Stéphane updated it, but I have not tried it in 2.12

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 12, 2017

Hello @jypeter
I have tried for point 1-3 ; but getting error as No boundary data is available for grid.

i dont have vaccum lib installed??

then using above mentioned self written code; tried to reformat data ; still not able to index it with lat / long fields as available in geos5_sample.nc.

Please have a look at my code and suggest some changes. today I have attached input file for more insight .

I am stuck so pls spare some time .

thanks

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 23, 2017

Hello @doutriaux1

Any update from your side?

pls help.

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 23, 2017

sorry. Dropped the ball on this one. WIll take a look today.

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 23, 2017

I have attached input file too.pls HV a look

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 23, 2017

@jaigsingla ok now i'm confused the file you're pointing out is on a regular lat/lon grid. i.e latitude and longitude are both 1D. You stated that your data is has latitude and lngitude as 2D arrays (180,360) which sounds like a curvilinear grid. In order to store them on a regular grid you will need to regrid them to such grid. In order to regrid you will need to know the vertices. Please see again Notebook here in order to apply the curvilinear and regrid the data.

@jaigsingla
Copy link
Author

@jaigsingla jaigsingla commented Oct 24, 2017

Sample.zip is the filename I am referring to. I hope you have gone through. Here latitude , longitude and parameter is stored differently. Now how to rearrange these to a desirable grid

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 24, 2017

@jaigsingla again you have data on a curvilinear grid. You need to know the vertices/bounds of each lat/lon in order to regrid. Can you infer it from the centers? I (and the software) DO NOT have information to know this. @dnadeau4 @painter1 do you guys know if there is a CF compliant way to store such information (figure out the bounds from the centers)?

@doutriaux1
Copy link
Contributor

@doutriaux1 doutriaux1 commented Oct 24, 2017

@durack1 please chime on this.

@dnadeau4
Copy link
Contributor

@dnadeau4 dnadeau4 commented Feb 13, 2018

@jaigsingla I will close this issue, you can re-open it if you need more information.

@dnadeau4 dnadeau4 closed this Feb 13, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants