LSEG_NLDAS_MAP File Format #247
Comments
|
Hey @alexwlowe @katealbi11 @kylewlowe @snhandelsman -- I just looked through the code and started mapping out the basics of the file to the best of my knowledge. If someone is game to look into this while we wait to hear from Gopal, I am available for email tech support. That is not to say we don't have plenty to do documenting, downloading the data, figuring out the coverages outside of Virginia, etc. |
|
That's interesting @rburghol! Do you know what format the LandSeg parameter is named? Does each land seg have unique names, codes, numbers, etc? Also, are the land segments the same as the cbp phase 6 land segs? |
|
No clue as to the format :), but I put in an email to Gopal. As for the general concept, the land segments should absolutely correspond to phase 6 land segments, just that, we don't year quiteknow the answer there. What I can see, I think, is that there is no weighting information, as the code appears to simply consider each overlapping cell equally. If you were bored and wanted to try your hand at creating a config file go ahead! (you'd have to figure out how to reference the NLDAS2 grid cell based on what ou can infer from the code), |
|
Take 1 at taking a stab at getting I created the config file mimicking the format above and called it LandSegNLDASFile: Based on the error, I'm assuming I messed up somewhere before LandSegNLDASFile was even read, I will do some looking in the function to try to trouble shoot. Update: I believe the error occurred at line 101. It looks like the DPT file is created on line 91 using LandSeg, which I believe ties into the LandSegNLDASFile, so I think it actually is that file that is breaking it. |
|
It is saying it can't open ./1991010100-1991010123/N51029.DPT - Could the problem be that the directory does not exist? In other words, if you created a directory named "1991010100-1991010123" maybe it would work? |
|
Thanks @rburghol! That helped a lot, past that error! Take 2 with actual progress |
|
Take 3 The function created 6 files titled N51029.(three letter code) which is I think formatted <Landseg.Metric code> Update: If you run 2 land segments and list both land segments, all the outputs in the files turn into zeros. If you say 2 land segments but only list one of them (following config file), it only creates files for the land seg you specified, and the values are the same as if you were to say there was only 1 land segment. |
|
This looks super encouraging!! By what I can see here, I think we can avert the error that you got when specifying multiple landsegs, just create a config file for each land seg, and be done with it. That is kid of a clunky workaround, but not the worst thing ever done in the history of data processing :). |
|
I tried this from your screen capture in the /backup/meteorology directory, and got close. But, the data in /backup/meteorology is not quite the same as that in Alex's home directory. But clearly you know how to make it work, so I think we're on the case! For reference and future copy-pasteable docs: |
|
Hey Rob, Since there is currently only the raw NLDAS data in the meteorology directory, you are going to have to run the GRIB to ASCII function first I believe this will work And the lines in LSEG_NLDAS_MAP should be |
|
Thanks! Sorry to ask a question before I read the other doc fully! |
|
Alex - I have gone through this a bunch of times now, and we have some small tweak that needs to occur:
Try #1: Fails with Try 2: Succeeds from Alex's home directory. So, I conclude that there is something in your directory, maybe in the "-R" folder? That appears to be the only place where the file `` exists. Looping @jdkleiner in on this as I will be out next week -- I won't be totally unreachable, but I'll need his help. |
|
Hey Rob, There are two reasons why this may not have worked anywhere but my home directory.
|
|
1. I *appear* to have successfully ran NLDAS_GRIB_to_ASCII
2. I *definitely* had a copy of the LSEG_NLDAS_MAP file in the directory I
was in
What I suspect is that something in "-R..." directory may br required?
On Mon, Jun 28, 2021 at 9:33 AM alexwlowe ***@***.***> wrote:
Hey Rob,
There are two reasons why this may not have worked anywhere but my home
directory.
1.
The timeseries data that is extracted form NLDAS2_GRIB_to_ASCII needed
to run the NLDAS2_ASCII_to_LSegs is only in my directory at the
moment. We have not batch run this function for all of the grids/timeframe
we need yet because we were still downloading the raw NLDAS data.. Based on
the error message you received, I think this is the reason.
2.
The LSEG_NLDAS_MAP config file is only in my home directory. If you
did not create a copy of the file wherever you were running the function,
it probably wouldn't work. I did not test to see if it works to run the
function in a different directory than the file, but I would assume it
needs to be in the same directory unless you specify a path.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#247 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABC4AIVQQYPV2RC6UHTOROLTVB2ZPANCNFSM47FZIYLA>
.
--
--
Robert W. Burgholzer
'Making the simple complicated is commonplace; making the complicated
simple, awesomely simple, that's creativity.' - Charles Mingus
|
|
That's interesting Rob, I'll do some more testing and looking around my directories today to try and figure out what's up. I'll update this comment with what I find. Update: I was able to run the function in the /backup/meteorology directory for Jan 1st, 2017 using the following commands. I think that you have to make sure that the time series directory (./out/grid_met_csv) has the timeseries data for the grid you specify in the config folder, and that the land seg directory (./out/lseg_csv) has a directory within it that outlines the timeframe that the function is running (2017010100-2017010123 in this case). I also think the LSEG_NLDAS_MAP config file needs to be in your working directory (/backup/meteorology), or you have to specify it's path in the function. Hopefully this helps! |
|
Area weighting update: It appears that the To test this hypothesis, I used Gopal's example file to run Hopefully this all makes sense, I know sometimes my writing isn't completely coherent! |
|
I think we need to make sure someone else can run the script from /backup/meteorology to verify. |
|
We were all able to run |
|
Quick Note: The function seems to break after the first 168 grid IDs from the config file have been run. The error message says that the grid .txt file cannot be opened, even if the .txt file for the grid is in the correct directory and has data. I do not know if this is just because the function can only crunch so many files worth of numbers at one time or if there is possibly a way around the issue. If not, then the function can just be run multiple times with separate config files until all land segments and corresponding grids are accounted for. p.s. there is now a config file called LSEG_NLDAS_ALL located in the /backup/meteorology directory that contains a list of all of the land segments in VA's basins and their corresponding grid IDs! It is in the correct format to run the |
File is text, with 1 format parameter per line. Note, in the c++ code, the signifier ">>" loads the next line of the file into the given variable, for example, the line
LandSegNLDASFile >> NumLandSeg;reads the next line from the fileLandSegNLDASFile, and puts the value into the integer variableNumLandSeg. By looking through the code for each occurence of ">>", we can map our the contents of the config file.The text was updated successfully, but these errors were encountered: