Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crefl2gtiff.sh fails when running in realtime #99

Closed
kathys opened this issue Apr 21, 2014 · 5 comments
Closed

crefl2gtiff.sh fails when running in realtime #99

kathys opened this issue Apr 21, 2014 · 5 comments
Assignees
Labels

Comments

@kathys
Copy link
Collaborator

kathys commented Apr 21, 2014

Dave, I got an email from one of our great users and beta testers. She is a great help to us. We had this email exchange today, which includes one of the log files from a failed execution. Any ideas?

From: Anikó KERN anikoc@nimbus.elte.hu
Subject: Re: IMAPP RealEarth Web Mapping Service for GeoTIFF Satellite Image Display Beta Software Available
Date: April 21, 2014 at 3:45:33 PM CDT
To: Kathy Strabala kathys@ssec.wisc.edu

Dear Kathy,

I'm so sorry for my description :)
Yes, during the real-time and automatic processing of the overpasses the process of modis2gtiff fails sometimes, but a few hours later when I generate the Level1b HDF files again with SeaDAS and start to use the Polar2Grid again it works correctly.

I send you now the log file.

Thank you for your help!
Aniko

Kathy Strabala írta:
Aniko,
I am not sure that I completely understand your problem.  When you run in real-time polar2grid does not work, but when you go back and reprocess it does work?  When the software fails, it leaves behind a .log file with alot of information in it. Can you please send us one of these log files, along with the exact command you using?
Thanks,
Kathy

On Apr 21, 2014, at 3:19 PM, Anikó KERN anikoc@nimbus.elte.hu wrote:
Dear Kathy,

Thank you for your kind attention and the detailed answer :), these are good news.

Would you mind, if I asked something again?

Now the Polar2grid and the WMS is running automatically for our new overpasses (which means that when there are new available Level1b HDF files then GeoTIFF files will be generated and then uploaded). But it happened in some cases that the process failed during the running of the modis2gtiff for daytime overpasses. The message was the following yesterday:


Running modis2gtiff.sh -f a1.14110.1232.250m.hdf…
INFO     : Using default grid configuration: 'grids.conf'
INFO     : Loading grid configuration 'grids.conf'
INFO     : Loading package provided rescale config: 'rescale_configs/rescale_inc.8bit.conf'
INFO     : Extracting swaths...
INFO     : Getting data file info...
INFO     : Creating binary files for latitude and longitude data
INFO     : Creating binary files for image data
INFO     : Determining what grids the data fits in...
INFO     : Running ll2cr for grid wgs84_fit and bands [('visible', '02'), ('visible', '01')]
INFO     : row and column calculation complete
ERROR    : Couldn't find fornav output file 'result_geo_250m_nav_visible_02_wgs84_fit.real4.9999999999999999455752309870428160.9999999999999999455752309870428160'
WARNING  : fornav failed for geo_250m_nav band, grid wgs84_fit, remapping as reflectance
WARNING  : Cleaning up for this job...
ERROR    : Removing visible02 because of bad fornav execution
ERROR    : Removing visible01 because of bad fornav execution
ERROR    : The last grid job for grid wgs84_fit was removed
ERROR    : Fornav was not able to complete any remapping for navigation set geo_250m_nav
ERROR    : Remapping data failed
ls: cannot access /home/modis/polar2grid/data/_vis_tif: No such file or directory

/home/modis/polar2grid/data/modis2gtiff_20140420_123200.log

When I realized it, I started the process again, but from the beginning, generating the Level1b HDF files again with the SeaDAS software. Then the Polar2grid was working perfectly. (I have these two datasets (real-time and postprocessed HDF files), and I tested it again.) I found this until now only in cases of Aqua overpasses, but not for all overpasses of course. I use the same SeaDAS software both for the automatic and manual running.

And, one more thing, the same script with the *1000m.hdf (I mean for example: modis2gtiff.sh -f a1.14110.1232.1000m.hdf) does not show this problem. Neither the crefl2gtiff.sh script (eg. in this way: a1.14110.1232.crefl.{1000m,250m}.hdf). These are working fine in every cases.

Do you think it is connected to the real-time generating of the Level1b files, or do I something wrong?

I’m really sorry for my question and to take your time again. Again and always :) And I hope the answer is simple :)

Aniko

Kathy Strabala írta:
Hi Aniko,
We are getting ready to release the official version of this product, and I realized that I had not answered all of your questions.
Nevertheless I believe that it is a great software, you made again something fascinating. I love DBGE as well, I check our overpasses every day in that application, but this new one is really what I missed :) Can we see in the future the level2 products also in this new application? :)
We really hope to release a new version of polar2grid in the future that is able to display some of the level 2 products.
Only one last thing came into my mind. I used to look at the well known false color images of bands 3-6-7 (where one can easily distinguish the snow cover by the clouds) and the 7-2-1 (which helps to identify areas affected by floods). Is there any possibility to implement them later to a newer version of this software?
We also hope that future version of polar2grid will allow users to select the bands they want to use to create RGB images.
Finally, we think we fixed the problem you showed us about Aqua passes crossing the 00 UTC date/time line and not being composited together.  Once we release the software, you  will have to  upgrade  to a new VM and copying and using new wmsupload and imapp-upload scripts to fix this.  I hope the re-installation goes more easily than the original installation.
Thanks again,
Kathy
Kathleen Strabala
Assistant Scientist
University of Wisconsin - Madison
Cooperative Institute for Meteorological Satellite Studies (CIMSS)
1225 West Dayton Street
Madison, WI   53706
Phone: (608) 263-8752
Fax: (608) 262-5974
kathy.strabala@ssec.wisc.edu
On Mar 30, 2014, at 8:33 AM, Anikó KERN anikoc@nimbus.elte.hu wrote:
Dear Kathy,

It might easily happen that I'm the last one who tested your beta software, very sorry for the delay, but we had many technical problems with the virtual machine.

We are still not ready with testing it, but I know that you are waiting for our results, therefore I send you what we learned until now. Our system administrator is still working on this adaptation, but he just wrote me that he had closed himself out from our Linux server remotely :) while he was checking the firewall rules needed to use the port 8001. It means that we can continue to adapt and test the software only tomorrow, sitting down in front of that machine physically.

Do you plan that this software should be installed on any kind of Linux? I understand from the manual that we should have RedHat or CentOS Linux to install the VM, but there are many other Linux distributions out there. Does the other DB stations these systems as well?

We think that the present form of this software is appropriate really only for your systems (RedHat or CentOS). However our administrator is seemed to solve every problem arose until, but as he told us it was not simple at all. He started to overwrite the installation script, but then he had to finish the installation manually, therefore we can not give you any alternative script concerning to the Debian system. In any case, he told me that there are a few files that are located in different directories in our Debian Linux, and there are other tricks that he applied.

Now our VM is working, but we can not still check the port 8001 because of some issues about the firewall (what led to the closing out of every remote user right now :))

The remarks which I started to note during the fist steps of the installation are the followings, but all was concerning the differences originated from the different Linux systems.
For example the command ‘yum’ is used (might be only) on Fedora Linux, but on other systems it can be else, eg. on Debian Linux (what we are using) it is apt-get.
We had problems with the Libvirtd, because on Debian it has other place and name than it was in the install script. Our administrator modified the script install_realearth-imapp.sh to be compatible with Debian (before he decided to do it manually). We had another problem with Virsh as well, because it turned out that the switch net-update it is not found in our Debian system, therefore our administrator upgraded our full Debian system. There is a question during the installation, the last one: regarding the dataroot (NFS mount point). I think it can not be modified after the installation, but I was not sure which directory should I choose. I think it would be good if it was explained in the documentation before the installation.

Nevertheless I believe that it is a great software, you made again something fascinating. I love DBGE as well, I check our overpasses every day in that application, but this new one is really what I missed :) Can we see in the future the level2 products also in this new application? :)

Only one last thing came into my mind. I used to look at the well known false color images of bands 3-6-7 (where one can easily distinguish the snow cover by the clouds) and the 7-2-1 (which helps to identify areas affected by floods). Is there any possibility to implement them later to a newer version of this software?

As soon as we have the total success with this application, I will write you.

I’m sorry again for my (and our administrator’s) speed :)

Aniko

Kathy Strabala írta:
Dear Colleagues,
The University of Wisconsin-Madison Space Science and Engineering Center (SSEC) has prepared a software package as part of the International MODIS/AIRS Processing Package (IMAPP) that enables users to display satellite imagery in a Google Maps or Google Earth interface.  It is distributed as a virtual machine (VM) that is capable of displaying MODIS (and VIIRS) GeoTIFF images created from the IMAPP/CSPP Polar2grid software package in support of Direct Broadcast users.   IMAPP RealEarth is a browser-based GeoTIFF imagery viewer powered by SSEC’s RealEarth Web Mapping System.  IMAPP RealEarth allows users to process imagery generated by their IMAPP installation for display in Google Maps, and share that imagery with users through a web mapping service.  The following define the host system requirements.  Please note: Your system will not be able to support the IMAPP RealEarth Virtual Machine (VM) unless it meets all of these requirements:

  • Intel or AMD CPU with 64-bit instruction support
  • RedHat or CentOS Linux v.6 host server
  • Root privileges on host server
  • Hardware virtualization enabled in the host system BIOS
    The web interface allows users to select from 47 different preset Aqua and Terra MODIS and/or VIIRS band swath or composite displays.  Which bands and how they are displayed is configurable.  Users can add their own (non Polar2grid) GeoTIFF files if desired.  Users also have the capability to share their display with  colleagues and friends or export it to Google Earth.
    The IMAPP RealEarth Beta installation and execution instructions are available here:
    http://wms.ssec.wisc.edu:8000/imapp/IMAPP_RealEarth_Web_Mapping_Service_V1.0_Beta.pdf
    Please let us know if you have any questions or comments.  We look forward to your feedback.
    The IMAPP/CSPP Polar2Grid software is a package for reprojecting MODIS Level-1B (L1B) files and VIIRS Science Data Records (SDRs) files into GeoTIFF format.  It is freely distributed through this website:
    http://cimss.ssec.wisc.edu/imapp/modis_polar2grid_v1.1.shtml
    An example of an implementation of the RealEarth software centered over the US can be viewed here:  http://imapp.ssec.wisc.edu:8001/
    Best Regards,
    Kathy (for the IMAPP RealEarth Team)
    Kathleen Strabala
    Assistant Scientist
    University of Wisconsin - Madison
    Cooperative Institute for Meteorological Satellite Studies (CIMSS)
    1225 West Dayton Street
    Madison, WI   53706
    Phone: (608) 263-8752
    Fax: (608) 262-5974
    kathy.strabala@ssec.wisc.edu mailto:kathy.strabala@ssec.wisc.edu
    [2014-04-21 22:41:24,600] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'mod07_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.mod07.hdf'
    [2014-04-21 22:41:24,600] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty nav file set 'mod07_nav'
    [2014-04-21 22:41:24,600] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.inversion.hdf'
    [2014-04-21 22:41:24,600] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.ist.hdf'
    [2014-04-21 22:41:24,600] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.ndvi.1000m.hdf'
    [2014-04-21 22:41:24,600] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.modlst.hdf'
    [2014-04-21 22:41:24,600] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.mask_byte1.hdf'
    [2014-04-21 22:41:24,601] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.icecon.hdf'
    [2014-04-21 22:41:24,601] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.1000m.hdf'
    [2014-04-21 22:41:24,601] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'geo_1000m_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.mod28.hdf'
    [2014-04-21 22:41:24,601] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty nav file set 'geo_1000m_nav'
    [2014-04-21 22:41:24,601] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty file pattern 'mod06_nav':'[at]1.\d\d\d\d\d.\d\d\d\d.mod06ct.hdf'
    [2014-04-21 22:41:24,601] : PID  32285 : DEBUG    : polar2grid.modis.modis_guidebook : sort_files_by_nav_uid : Removing empty nav file set 'mod06_nav'
    [2014-04-21 22:41:24,601] : PID  32285 : DEBUG    : main : run_glue : Calling geo_250m_nav navigation set
    [2014-04-21 22:41:24,609] : PID  32285 : DEBUG    : main : run_glue : Waiting for subprocess for geo_250m_nav navigation set
    [2014-04-21 22:41:24,609] : PID  32294 : DEBUG    : main : process_data_sets : Processing geo_250m_nav navigation set
    [2014-04-21 22:41:24,610] : PID  32294 : INFO     : polar2grid.grids.grids : init : Using default grid configuration: 'grids.conf'
    [2014-04-21 22:41:24,610] : PID  32294 : INFO     : polar2grid.grids.grids : init : Loading grid configuration 'grids.conf'
    [2014-04-21 22:41:24,675] : PID  32294 : INFO     : polar2grid.core.roles : load_config : Loading package provided rescale config: 'rescale_configs/rescale_inc.8bit.conf'
    [2014-04-21 22:41:24,676] : PID  32294 : INFO     : main : process_data_sets : Extracting swaths...
    [2014-04-21 22:41:24,851] : PID  32294 : INFO     : polar2grid.modis.modis_to_swath : make_swaths : Getting data file info...
    [2014-04-21 22:41:24,852] : PID  32294 : INFO     : polar2grid.modis.modis_to_swath : make_swaths : Creating binary files for latitude and longitude data
    [2014-04-21 22:41:25,267] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Variable Latitude is using scale value None and offset value None
    [2014-04-21 22:41:25,267] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Interpolating to higher resolution: Latitude
    [2014-04-21 22:41:33,124] : PID  32294 : DEBUG    : polar2grid.core.fbf : append : 21440 rows in output file
    [2014-04-21 22:41:33,609] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : After loading, variable Latitude has fill value -999.0 and data range -999.0 to 35.9056
    [2014-04-21 22:41:34,003] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Variable Longitude is using scale value None and offset value None
    [2014-04-21 22:41:34,003] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Interpolating to higher resolution: Longitude
    [2014-04-21 22:41:42,395] : PID  32294 : DEBUG    : polar2grid.core.fbf : append : 21440 rows in output file
    [2014-04-21 22:41:42,904] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : After loading, variable Longitude has fill value -999.0 and data range -999.0 to 27.4015
    [2014-04-21 22:41:42,929] : PID  32294 : INFO     : polar2grid.modis.modis_to_swath : make_swaths : Creating binary files for image data
    [2014-04-21 22:41:46,173] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Clipping saturation values
    [2014-04-21 22:41:46,640] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Variable EV_250_RefSB is using scale value 2.7127638532e-05 and offset value -0.0
    [2014-04-21 22:41:50,779] : PID  32294 : DEBUG    : polar2grid.core.fbf : append : 21440 rows in output file
    [2014-04-21 22:41:51,467] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : After loading, variable EV_250_RefSB has fill value 65535 and data range 0.00547978 to 1.77778
    [2014-04-21 22:41:55,766] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Clipping saturation values
    [2014-04-21 22:41:56,231] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : Variable EV_250_RefSB is using scale value 4.93939332955e-05 and offset value -0.0
    [2014-04-21 22:42:00,103] : PID  32294 : DEBUG    : polar2grid.core.fbf : append : 21440 rows in output file
    [2014-04-21 22:42:00,770] : PID  32294 : DEBUG    : polar2grid.modis.modis_to_swath : _load_data_to_flat_file : After loading, variable EV_250_RefSB has fill value 65535 and data range 0.0155097 to 3.23698
    [2014-04-21 22:42:00,807] : PID  32294 : DEBUG    : main : process_data_sets : band_info after prescaling: [('visible', '02'), ('visible', '01')]
    [2014-04-21 22:42:00,807] : PID  32294 : INFO     : main : process_data_sets : Determining what grids the data fits in...
    [2014-04-21 22:42:00,807] : PID  32294 : DEBUG    : polar2grid.grids.grids : create_grid_jobs : Kind visible Band 02 can handle these grids: ''any_proj4_grid''
    [2014-04-21 22:42:00,807] : PID  32294 : DEBUG    : polar2grid.grids.grids : create_grid_jobs : Kind visible Band 01 can handle these grids: ''any_proj4_grid''
    [2014-04-21 22:42:00,807] : PID  32294 : DEBUG    : polar2grid.grids.grids : create_grid_jobs : Kind visible band 02 will be remapped to grid wgs84_fit
    [2014-04-21 22:42:00,808] : PID  32294 : DEBUG    : polar2grid.grids.grids : create_grid_jobs : Kind visible band 01 will be remapped to grid wgs84_fit
    [2014-04-21 22:42:01,262] : PID  32294 : INFO     : polar2grid.remap : run_ll2cr : Running ll2cr for grid wgs84_fit and bands [('visible', '02'), ('visible', '01')]
    [2014-04-21 22:42:02,156] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Data west longitude: -743.735535
    [2014-04-21 22:42:02,496] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Data east longitude: 27.401478
    [2014-04-21 22:42:02,834] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Data south latitude: -743.355225
    [2014-04-21 22:42:03,174] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Data upper latitude: 71.335548
    [2014-04-21 22:42:03,176] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Grid Corners: 1000000000000000019884624838656.000000,-0.786251
    [2014-04-21 22:42:03,176] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Grid Origin: -3.141593,1000000000000000019884624838656.000000
    [2014-04-21 22:42:03,176] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Grid width/height (9999999999999999455752309870428160, 9999999999999999455752309870428160)
    [2014-04-21 22:42:03,180] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Real upper-left corner (-180.000000,57295779513082318043087410036736.000000)
    [2014-04-21 22:42:03,180] : PID  32296 : DEBUG    : polar2grid.ll2cr : ll2cr : Real lower-right corner (8063664102031684.000000,0.000000)
    [2014-04-21 22:42:29,995] : PID  32296 : INFO     : polar2grid.ll2cr : ll2cr : row and column calculation complete
    [2014-04-21 22:42:33,611] : PID  32294 : DEBUG    : polar2grid.core.fbf : check_stem : Checking stem 'result_geo_250m_nav_visible_02_wgs84_fit'
    [2014-04-21 22:42:33,612] : PID  32294 : DEBUG    : polar2grid.core.fbf : check_stem : Checking stem 'result_geo_250m_nav_visible_01_wgs84_fit'
    [2014-04-21 22:42:33,613] : PID  32301 : DEBUG    : polar2grid.ms2gt : fornav : Running fornav with 'fornav 2 -t f4 f4 -f 65535 65535 -F 65535 65535 -D 10 -d 1 -s 0 0 5416 536 40 ll2cr_geo_250m_nav_wgs84_fit_cols.real4.5416.21440 ll2cr_geo_250m_nav_wgs84_fit_rows.real4.5416.21440 image_geo_250m_nav_visible_02.real4.5416.21440 image_geo_250m_nav_visible_01.real4.5416.21440 1e+34 1e+34 result_geo_250m_nav_visible_02_wgs84_fit.real4.9999999999999999455752309870428160.9999999999999999455752309870428160 result_geo_250m_nav_visible_01_wgs84_fit.real4.9999999999999999455752309870428160.9999999999999999455752309870428160'
    [2014-04-21 22:42:34,944] : PID  32301 : ERROR    : polar2grid.ms2gt : fornav : Couldn't find fornav output file 'result_geo_250m_nav_visible_02_wgs84_fit.real4.9999999999999999455752309870428160.9999999999999999455752309870428160'
    [2014-04-21 22:42:35,014] : PID  32294 : WARNING  : polar2grid.remap : run_fornav : fornav failed for geo_250m_nav band, grid wgs84_fit, remapping as reflectance
    [2014-04-21 22:42:35,014] : PID  32294 : DEBUG    : polar2grid.remap : run_fornav : Exception was:
    Traceback (most recent call last):
     File "/home/modis/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-1.1.6-py2.7.egg/polar2grid/remap.py", line 305, in run_fornav
       fornav_dict = fornav_job["result"].get()
     File "/home/modis/polar2grid/ShellB3/lib/python2.7/multiprocessing/pool.py", line 528, in get
       raise self._value
    RuntimeError: Couldn't find fornav output file 'result_geo_250m_nav_visible_02_wgs84_fit.real4.9999999999999999455752309870428160.9999999999999999455752309870428160'
    [2014-04-21 22:42:35,036] : PID  32294 : WARNING  : polar2grid.remap : run_fornav : Cleaning up for this job...
    [2014-04-21 22:42:35,036] : PID  32294 : ERROR    : polar2grid.remap : run_fornav : Removing visible02 because of bad fornav execution
    [2014-04-21 22:42:35,036] : PID  32294 : ERROR    : polar2grid.remap : run_fornav : Removing visible01 because of bad fornav execution
    [2014-04-21 22:42:35,036] : PID  32294 : ERROR    : polar2grid.remap : run_fornav : The last grid job for grid wgs84_fit was removed
    [2014-04-21 22:42:35,036] : PID  32294 : ERROR    : polar2grid.remap : run_fornav : Fornav was not able to complete any remapping for navigation set geo_250m_nav
    [2014-04-21 22:42:35,036] : PID  32294 : DEBUG    : main : process_data_sets : Remapping Error:
    Traceback (most recent call last):
     File "/home/modis/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-1.1.6-py2.7.egg/polar2grid/modis2gtiff.py", line 151, in process_data_sets
       do_single_sample=fornav_m,
     File "/home/modis/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-1.1.6-py2.7.egg/polar2grid/remap.py", line 361, in remap_bands
       do_single_sample=do_single_sample)
     File "/home/modis/polar2grid/ShellB3/lib/python2.7/site-packages/polar2grid-1.1.6-py2.7.egg/polar2grid/remap.py", line 324, in run_fornav
       raise RuntimeError("Fornav was not able to complete any remapping for navigation set %s" % (nav_set_uid,))
    RuntimeError: Fornav was not able to complete any remapping for navigation set geo_250m_nav
    [2014-04-21 22:42:35,037] : PID  32294 : ERROR    : main : process_data_sets : Remapping data failed
    [2014-04-21 22:42:35,038] : PID  32285 : DEBUG    : main : main : Processing returned status code: 12
@djhoese
Copy link
Member

djhoese commented Apr 21, 2014

Hi Kathy,
This sounds familiar to a problem I was having with my most recent development version (for the new version 1.2 release), but this shouldn't be happening in the older versions of polar2grid. It looks like there may be some fill values or bad data getting through. You can see from the log that when it is calculating the grid locations (ll2cr) it says that the west and south locations are:

[2014-04-21 22:42:02,156] : PID 32296 : DEBUG : polar2grid.ll2cr : ll2cr : Data west longitude: -743.735535

The best way for me to solve this is to look at the data files. I don't have any good guesses as to the exact problem, but given that this user said 1000m files and 1000m and 250m crefl files worked make me think that its in the code that interpolates the 1000m resolution navigation to 250m resolution. Let me know if it is easy enough to get me the data otherwise I will come up with another debug method.

@djhoese djhoese self-assigned this Apr 21, 2014
@djhoese djhoese added the bug label Apr 21, 2014
@kathys
Copy link
Collaborator Author

kathys commented Apr 22, 2014

Aniko staged the data sets that she was using:

ear Kathy,

It would be very kind of Dave, if he could check these HDF files, I'm really grateful to him. Here are the files with your naming convention:

ftp://ftp.ssec.wisc.edu/pub/incoming/a1.14110.1232.250m.hdf
ftp://ftp.ssec.wisc.edu/pub/incoming/a1.14110.1232.500m.hdf
ftp://ftp.ssec.wisc.edu/pub/incoming/a1.14110.1232.1000m.hdf
ftp://ftp.ssec.wisc.edu/pub/incoming/a1.14110.1232.geo.hdf

This overpass was normal, without any problem:
http://nimbus.elte.hu/kutatas/sat/latest/mds/images/browse/1KM_AQUA_63626_20140420_123213.xxxxx.jpg

Thank you and Dave very much in advance,
Aniko

@djhoese
Copy link
Member

djhoese commented Apr 22, 2014

It looks like the navigation data has fill values in it (-999.0). The code that interpolates the 250m geolocation from the 1000m was derived from IDL code that didn't handle fill values...at all. Pretty big oversight on my part.

So what is happening is that the fill values are being used during interpolation. Let's say 2 pixels are -999.0 and -28.0 (a valid longitude). The pixels being created then have values like -743, -488, and -233. When trying to discover the bounding box of the swath the code is just checking the minimum for the whole swath (excluding the -999.0 fill value) and is getting -743 back. This is creating huge/incorrect grids as output.

I will let you know if/when I have a fix for this. This should only effect the 250m resolution products. A quick fix that might work would be to define a static grid. This way the lat/lon bounding box is never used by the remapping, but you would still have a few pixels in the wrong locations.

@djhoese
Copy link
Member

djhoese commented Apr 22, 2014

I think I've found a solution. Here is an example of the output image:

aqua_modis_visible_01_20140420_123200_wgs84_fit

I can probably pull the fix into the 1.1 release or we can talk about including the v1.2 with RealEarth soon. Email me or comment here or we can talk tomorrow at our normal meeting.

djhoese added a commit that referenced this issue Apr 22, 2014
@kathys
Copy link
Collaborator Author

kathys commented Apr 22, 2014

Hi Dave,

Let’s talk about it tomorrow. Spent the day trying to figure out what happened to our operational system overnight.

Kathy

On Apr 22, 2014, at 11:27 AM, David notifications@github.com wrote:

I think I've found a solution. Here is an example of the output image:

I can probably pull the fix into the 1.1 release or we can talk about including the v1.2 with RealEarth soon. Email me or comment here or we can talk tomorrow at our normal meeting.


Reply to this email directly or view it on GitHub.

@djhoese djhoese closed this as completed Jun 12, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

2 participants