can't access 3d data from MERRA2 openDAP server

[bucricket]

Hi All,

I have tried to work with MERRA2 openDap team with this problem but they say the issue is not on their end. Basically I can access 2d data but not 3d data. Please see the code below.

from pydap.client import open_url
from pydap.cas.urs import setup_session
session = setup_session(earthLoginUser, earthLoginPass)

fullUrl4 = "https://goldsmr4.gesdisc.eosdis.nasa.gov/opendap/MERRA2/M2I1NXASM.5.12.4/2015/08/MERRA2_400.inst1_2d_asm_Nx.20150802.nc4"
d = open_url(fullUrl4, session=session)
var = d['lat']
var[0:10]

I get the data
array([-90. , -89.5, -89. , -88.5, -88. , -87.5, -87. , -86.5, -86. , -85.5])

however if I try to access the 3d data it fails.

fullUrl5 = "https://goldsmr5.gesdisc.eosdis.nasa.gov/opendap/hyrax/MERRA2/M2I3NVASM.5.12.4/2015/08/MERRA2_400.inst3_3d_asm_Nv.20150802.nc4"
d = open_url(fullUrl5, session=session)
var = d['lat']
var[0:10]

I get the following error:

Traceback (most recent call last):

File "", line 1, in
var[0:10]

File "/Applications/anaconda/lib/python2.7/site-packages/pydap/model.py", line 258, in getitem
return self.data[index]

File "/Applications/anaconda/lib/python2.7/site-packages/pydap/handlers/dap.py", line 135, in getitem
dds, data = r.body.split(b'\nData:\n', 1)

ValueError: need more than 1 value to unpack

Any help would be greatly appreciated!

[laliberte]

I've conducted some tests and PR #57 appears to be fixing this issue on my end. @jameshiebert, any chance PR #57 could be merged?

[bucricket]

Hi @jameshiebert if @laliberte fixes the problem can you merge it? I'm really depending on pydap to get MERRA2 data. I can't find a work around right now.

Thanks!

[laliberte]

@bucricket, you can always pull PR #57 and install this patched up version of pydap and check if it works for you.

[bucricket]

yeah.... kind of new with installing with git. I'm trying to build a program with conda build. if you are familiar with conda build do you know how I could incorporate your fix?

I really appreciate your help!

[laliberte]

I think the best thing to do is to vendor pydap. This way you have complete control on the distribution and you can apply your own fixes. One way is to fork and then include your fork as a submodule in git. It is a bit confusing at first but it will give you much more freedom in the future. You can always keep your fork up-to-date with new commits to the main pydap repo.

I am working on my own fork that attempts to more continuously incorporates patches as I need them. You can see how a fork can be included in a project here.

Good luck!

[jameshiebert]

Hi guys. I'm just working through as many of the PRs as I can today. Thanks for your patiences and your active involvement. Are you both on the Pydap e-mail list? I'd like to have a conversation about how we can re-work Pydap's development workflow. I'm really excited about all the interest that has been expressed this last year, and it would be nice if we can all work out a better process that better meets everyone's needs. Me reviewing a group of PRs every couple of months is way too slow for you guys :) And not fair to the effort and investment that you've made in the code. Will you guys chime in if I bring this up on the list?

[laliberte]

Yes, I will.

I am currently working on a big patch that incorporates all of my PRs plus some. In particular, it provides an alternative API that mimics the one from netCDF4-python. I would recommend that this should be discussed on the mailing list.

[jameshiebert]

I'll go ahead and close this with #57 merged. If it continues to be problematic, feel free to re-open.

[bucricket]

Thanks for you help. So if I fork the master now it will include your fix from #57 correct? Thanks

…

On Feb 16, 2017, at 12:02 PM, Frederic Laliberte ***@***.***> wrote: I think the best thing to do is to vendor pydap. This way you have complete control on the distribution and you can apply your own fixes. One way is to fork and then include your fork as a submodule in git. It is a bit confusing at first but it will give you much more freedom in the future. You can always keep your fork up-to-date with new commits to the main pydap repo. I am working on my own fork <https://github.com/laliberte/pydap/tree/netcdf4_pydap/master> that attempts to more continuously incorporates patches as I need them. You can see how a fork can be included in a project here <https://github.com/laliberte/netcdf4_pydap/tree/netcdf4_soft_links/master>. Good luck! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#63 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHdNEE22-1qPRv3LicniNkFOOgJXTqpcks5rdIFAgaJpZM4MA5-a>.

[laliberte]

It should and it should work. For URS urls, you should use the check_url option.