New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Submission: MODIStsp #184

Closed
lbusett opened this Issue Jan 7, 2018 · 53 comments

Comments

Projects
None yet
6 participants
@lbusett
Contributor

lbusett commented Jan 7, 2018

Summary

  • What does this package do? (explain in 50 words or less):

Facilitates the creation of time series of raster images derived from MODIS Satellite Land Products by automating and standardizing several preprocessing steps (e.g., download, mosaicing, reprojection, resize, indexes computation, data extraction). Processing parameters are set using a powerful GUI. Non-interactive execution using saved Options Files allows updating time series when new images are available using cron jobs.

  • Paste the full DESCRIPTION file inside a code block below:
Package: MODIStsp
Title: A Tool for Automating Download and Preprocessing of MODIS Land Products
    Data
Type: Package
Version: 1.3.3.9000
Authors@R: c(person("Lorenzo", "Busetto", email = "lbusett@gmail.com", role = c("aut", "cre"),
    comment = c(ORCID = '0000-0001-9634-6038')), 
    person("Luigi", "Ranghetti", email = "ranghetti.l@irea.cnr.it", role = c("aut")))
Description: Allows automating the creation of time series of rasters derived
    from MODIS Satellite Land Products data. It performs several typical
    preprocessing steps such as download, mosaicking, reprojection and resize
    of data acquired on a specified time period. All processing parameters
    can be set using a user-friendly GUI. Users can select which layers of
    the original MODIS HDF files they want to process, which additional
    Quality Indicators should be extracted from aggregated MODIS Quality
    Assurance layers and, in the case of Surface Reflectance products
    , which Spectral Indexes should be computed from the original reflectance
    bands. For each output layer, outputs are saved as single-band raster
    files corresponding to each available acquisition date. Virtual files
    allowing access to the entire time series as a single file are also created.
    Command-line execution exploiting a previously saved processing options
    file is also possible, allowing to automatically update time series
    related to a MODIS product whenever a new image is available.
License: GPL-3
Depends:
    R (>= 3.1.3)
Imports:
    bitops (>= 1.0-6),
    data.table (>= 1.9.6),
    gdalUtils (>= 2.0.1.7),
    gWidgets (>= 0.0-54),
    gWidgetsRGtk2,
    methods,
    httr (>= 1.1.0),
    jsonlite,
    pacman,
    parallel,
    raster (>= 2.5-2),
    RCurl (>= 1.95-4.8),
    rgdal (>= 1.0-3),
    rgeos (>= 0.3-8),
    sp (>= 1.2-2),
    stringr (>= 1.0.0),
    xts (>= 0.9-7),
    xml2
Suggests:
    spelling,
    knitr,
    rmarkdown,
    png,
    grid,
    httptest,
    covr
SystemRequirements: Cairo >= 1.0.0, ATK (>= 1.10.0), Pango (>= 1.10.0), GTK+ (>=
    2.8.0), GLib (>= 2.8.0), Curl, GDAL (>= 1.6.3), PROJ.4 (>= 4.4.9)
URL: https://github.com/lbusett/MODIStsp
BugReports: https://github.com/lbusett/MODIStsp/issues
LazyData: true
VignetteBuilder: knitr
RoxygenNote: 6.0.1
Roxygen: list(markdown = TRUE)

  • URL for the package (the development repository, not a stylized html page):

    https://github.com/lbusett/MODIStsp

  • Please indicate which category or categories from our package fit policies this package falls under *and why(? (e.g., data retrieval, reproducibility. If you are unsure, we suggest you make a pre-submission inquiry.):

    • data retrieval, because the package facilitates access to MODIS satellite data;
    • geospatial data, because MODIS time series are widely used by the scientific community for several geospatial applications
  •   Who is the target audience and what are scientific applications of this package?  

    The target audience includes anyone involved/interested in monitoring the properties of the Earth surface through the analysis of time series of coarse resolution satellite imagery.
    MODIS satellite data are in fact used in many different scientific applications, since they allow global daily monitoring of the earth surface. Their applications are therefore too many to analyze in this context. Among the main ones, I can cite (borrowing a bit from wikipedia...) the monitoring of vegetation health by means of time-series analyses with vegetation indices, the analysis of long term land cover changes (e.g. to monitor deforestation rates), real time monitoring of snow/ice cover, analysis of water quality and of impacts of floodings, change of water levels of major lakes and the detection and mapping of wildland fires. MODIS data are also extensively used as ancillary input in modelling applications (e.g., biogeochemical models for Primary Production modelling; crop yield models). Additionally, MODIS data are often used by biologists/zoologists as ancillary data to monitor/model migratory patterns.

    (Introductory material concerning MODIS and its main applications can be found for example here and here)

  • Are there other R packages that accomplish the same thing? If so, how does
    yours differ or meet our criteria for best-in-category?

    The main “R” packages overlapping MODIStsp functionality are MODIS and MODISTools
    MODIStsp and MODIS have substantial overlaps, since they are both meant to provide access and preprocessing to MODIS products. Two aspects that in my opinion differentiate MODIStsp are its user-friendliness (I think that the MODIStsp Graphical User Interface is very helpful, in particular for less skilled "R" users, though I am clearly a bit biased on the topic), and its ability to allow on-the-fly creation of time series of standard/user provided Spectral Indexes (often used for time series analysis) and of MODIS quality indicators. On the other hand, MODIS provides some additional post-processing functionality (e.g., filtering/smoothing), which is something I'm planning to add in the near future.

    MODIStools is a very nice package, but it provides somewhat more limited functionality. It is based on the ORNL DAAC webservice (https://modis.ornl.gov/data/modis_webservice.html), and therefore provides access only to some MODIS products. Additionally, it is meant to allow access to data concerning a given set of points and their surroundings, while both MODIStsp and MODIS are meant to provide downloading capabilitites to the "full" MODIS raster data (or subsets of those, although the full HDFs need always to be downloaded).

    The ModisDownload function of package rts is also overlapping, but its functionalities are more limited than either MODIStsp and MODIS. (Actually, an early version of ModisDownload is what got us started in developing the package - proper acknwoledgement is given in MODIStsp documentation).

    Other packages providing access to MODIS data include MODISSnow (limited to snow cover products distributed by NSIDC) and modiscloud (which provides tools for processing Level 2 Cloud Mask SWATH products – not supported by MODIStsp)
    For further reference, our paper concerning MODIStsp published on CAGEO provides in Section 1 a short comparison of the different tools currently available for accessing MODIS data (both "R" based or not).

  •   If you made a pre-submission enquiry, please paste the link to the corresponding issue, forum post, or other discussion, or @tag the editor you contacted.

    #143

Requirements

Confirm each of the following by checking the box. This package:

  • does not violate the Terms of Service of any service it interacts with.

  • has a CRAN and OSI accepted license.

  • contains a README with instructions for installing the development version.

    NOTE ON INSTALLATION Installation (both from github or from CRAN) may be a bit problematic due the fact that MODIStsp requires the gWidgetsRGtk2 library to build its GUI, which often have installation problems on Windows and Mac due to requiring the GTK+ library, and needing several packages to be installed beforehand on Linux. In case of problems, it is better to install gWidgetsRGtk2 beforehand. Specific instructions are provided at (http://lbusett.github.io/MODIStsp/articles/installation.html)

  • includes documentation with examples for all functions.

  • contains a vignette with examples of its essential functions and uses.

  • has a test suite.

  • has continuous integration, including reporting of test coverage, using services such as Travis CI, Coveralls and/or CodeCov.

  • I agree to abide by ROpenSci's Code of Conduct during the review process and in maintaining my package should it be accepted.

Publication options

  • Do you intend for this package to go on CRAN?
    The package is already on CRAN (https://cran.r-project.org/package=MODIStsp)

  • Do you wish to automatically submit to the Journal of Open Source Software? If so:

    • The package has an obvious research application according to JOSS's definition.
    • The package contains a paper.md matching JOSS's requirements with a high-level description in the package root or in inst/.
    • The package is deposited in a long-term repository with the DOI:
    • (Do not submit your package separately to JOSS)
  • Do you wish to submit an Applications Article about your package to Methods in Ecology and Evolution? If so:

    • The package is novel and will be of interest to the broad readership of the journal.
    • The manuscript describing the package is no longer than 3000 words.
    • You intend to archive the code for the package in a long-term repository which meets the requirements of the journal.
    • (Please do not submit your package separately to Methods in Ecology and Evolution)

Detail

  • Does R CMD check (or devtools::check()) succeed? Paste and describe any errors or warnings:

  • Does the package conform to rOpenSci packaging guidelines? Please describe any exceptions:

  • If this is a resubmission following rejection, please explain the change in circumstances:

  • If possible, please provide recommendations of reviewers - those with experience with similar packages and/or likely users of your package - and their GitHub user names:

Michael Sumner - @mdsumner
Jeff Hanson - @jeffreyhanson
Leah Wasser - @lwasser
Robin Lovelace - @robin

@lwasser

This comment has been minimized.

lwasser commented Jan 8, 2018

hey @lbusett one of my students used this package in his final project in the fall! he showed it to me and said it was really nice.

@karthik

This comment has been minimized.

Member

karthik commented Jan 8, 2018

👋 @lbusett I will be handling your submission. Thanks also for the reviewer suggestions.
@lwasser Would you be interested in being one of the reviewers?

@karthik

This comment has been minimized.

Member

karthik commented Jan 8, 2018

I have also reached out to @jeffreyhanson by email.

@lbusett

This comment has been minimized.

Contributor

lbusett commented Jan 8, 2018

@karthik Great, thanks!
@lwasser Glad to hear about that!

@lwasser

This comment has been minimized.

lwasser commented Jan 8, 2018

sure @karthik i'll review this. i wanted to spend some time playing with it after my student showed it to me. What is the deadline? i'll need a few weeks given i just got back from travel today.

@jeffreyhanson

This comment has been minimized.

jeffreyhanson commented Jan 10, 2018

Sure, I'm happy to review this. I'm pretty busy at the moment though, so it might be a few weeks before I can get around to it if that's ok?

@karthik

This comment has been minimized.

Member

karthik commented Jan 10, 2018

Editor checks:

  • Fit: The package meets criteria for fit and overlap
  • Automated tests: Package has a testing suite and is tested via Travis-CI or another CI service.
  • License: The package has a CRAN or OSI accepted license
  • Repository: The repository link resolves correctly
  • Archive (JOSS only, may be post-review): The repository DOI resolves correctly
  • Version (JOSS only, may be post-review): Does the release version given match the GitHub release (v1.0.0)?

Editor comments

Work in progress (dealing with installation issues)


Reviewers: @lwasser and @jeffreyhanson
Due date: Feb 10, 2018

@karthik

This comment has been minimized.

Member

karthik commented Jan 10, 2018

@lwasser @jeffreyhanson Thanks very much! I've tentatively assigned Feb 10th. Please let me know if anything changes. 🙏

@lbusett

This comment has been minimized.

Contributor

lbusett commented Jan 10, 2018

@karthik Sorry to hear that you having problems in installation.... 😟 Unfortunately, as I reported in the submission, that is a common problem related to the requirements of the gWidgetsRGtk2 package. Instructions reported here: http://lbusett.github.io/MODIStsp/articles/installation.html are usually sufficient to solve the issue (unless you are working on a mac with os sierra, in which case it can be more dificult...).

Let me know if I can help in any way!

@sckott sckott added the package label Jan 16, 2018

@lbusett

This comment has been minimized.

Contributor

lbusett commented Jan 17, 2018

Hi @karthik . Just wanted to ask if in the end you managed to solve the installation issues or you are still having problems. In case you are working on a mac and you are still struggling, these are the "latest" instructions I found regarding how to install packages "RGtk2" (and consequently gWidgetsRGtk2) on mac os Sierra:

https://stackoverflow.com/a/46748516/6871135

@jeffreyhanson

This comment has been minimized.

jeffreyhanson commented Jan 31, 2018

Awesome work @lbusett! This package looks absolutely amazing. I've just started working on my review, but I thought I would let you know that some of the unit tests fail on my system in case you wanted to address this before I submit my review. I've included the output below and a copy of my session info.

> test()
Loading MODIStsp
Testing MODIStsp
✔ | OK F W S | Context
✔ |  4       | Check proper functioning of bbox_from_file [1.6 s]
✔ |  4       | get_yeardates works as expected when download_range == "Full"
✔ |  6       | get_yeardates works as expected when download_range == "Seasonal"
✔ |  7       | Check proper functioning of MODIStsp_addindex [0.2 s]
✔ | 21       | MODIStsp_extract works as expected [58.7 s]
⠏ |  0       | Check proper functioning of the main MODIStsp GUI
(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'

(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'

(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 87: Element 'markup' was closed, but the currently open element is 'span'

(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 87: Element 'markup' was closed, but the currently open element is 'span'

(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'

(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'

(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'

(R:19949): Gtk-WARNING **: Failed to set text from markup due to error parsing markup: Error on line 1 char 83: Element 'markup' was closed, but the currently open element is 'span'
✔ |  0       | Check proper functioning of the main MODIStsp GUI [4.1 s]
✔ |  6   2   | MODIStsp Test 0: Gracefully fail on input problems [14.9 s]
────────────────────────────────────────────────────────────────────────────────
test_MODIStsp.R:13: warning: Tests on MODIStsp
input string 1 is invalid in this locale

test_MODIStsp.R:18: warning: Tests on MODIStsp
input string 1 is invalid in this locale
────────────────────────────────────────────────────────────────────────────────
⠏ |  0       | MODIStsp Test 1: Basic processing on bands and quality
⠋ |  1       | MODIStsp Test 1: Basic processing on bands and quality
⠙ |  2       | MODIStsp Test 1: Basic processing on bands and quality
✔ |  2       | MODIStsp Test 1: Basic processing on bands and quality
        indicators [5.3 s]
✔ |  4       | MODIStsp Test 2: Geometric operations [3.2 s]
⠏ |  0       | MODIStsp Test 3: Computation of spectral indices and
            creation of time seriesWarning 1: Metadata exceeding 32000 bytes cannot be written into GeoTIFF. Transferred to PAM instead.
⠋ |  1       | MODIStsp Test 3: Computation of spectral indices and
⠙ |  2       | MODIStsp Test 3: Computation of spectral indices and
⠹ |  3       | MODIStsp Test 3: Computation of spectral indices and
⠸ |  4       | MODIStsp Test 3: Computation of spectral indices and
⠼ |  5       | MODIStsp Test 3: Computation of spectral indices and
            creation of time seriesWarning 1: Metadata exceeding 32000 bytes cannot be written into GeoTIFF. Transferred to PAM instead.
⠴ |  6       | MODIStsp Test 3: Computation of spectral indices and
⠦ |  7       | MODIStsp Test 3: Computation of spectral indices and
✔ |  7       | MODIStsp Test 3: Computation of spectral indices and
            creation of time series [22.0 s]
✖ |  0 1     | MODIStsp Test 4: HTTP download from NSIDC (seasonal) [5.8 s]
────────────────────────────────────────────────────────────────────────────────
test_MODIStsp.R:193: error: Tests on MODIStsp
argument is of length zero
1: MODIStsp(test = 4) at /home/jeff/GitHub/MODIStsp/tests/testthat/test_MODIStsp.R:193
2: MODIStsp_process(sel_prod = general_opts$sel_prod, start_date = general_opts$start_date, 
       end_date = general_opts$end_date, out_folder = general_opts$out_folder, out_folder_mod = general_opts$out_folder_mod, 
       reprocess = general_opts$reprocess, delete_hdf = general_opts$delete_hdf, sensor = general_opts$sensor, 
       download_server = general_opts$download_server, user = general_opts$user, password = general_opts$password, 
       https = prod_opts$http, ftps = prod_opts$ftp, start_x = general_opts$start_x, 
       start_y = general_opts$start_y, end_x = general_opts$end_x, end_y = general_opts$end_y, 
       full_ext = general_opts$full_ext, bbox = general_opts$bbox, out_format = general_opts$out_format, 
       out_res_sel = general_opts$out_res_sel, out_res = as.numeric(general_opts$out_res), 
       native_res = prod_opts$native_res, tiled = prod_opts$tiled, resampling = general_opts$resampling, 
       ts_format = general_opts$ts_format, compress = general_opts$compress, mod_proj_str = mod_proj_str, 
       outproj_str = general_opts$user_proj4, nodata_in = prod_opts$nodata_in, nodata_out = prod_opts$nodata_out, 
       rts = general_opts$rts, nodata_change = general_opts$nodata_change, scale_val = general_opts$scale_val, 
       scale_factor = prod_opts$scale_factor, offset = prod_opts$offset, datatype = prod_opts$datatype, 
       bandsel = general_opts$bandsel, bandnames = prod_opts$bandnames, indexes_bandsel = c(general_opts$indexes_bandsel), 
       indexes_bandnames = c(prod_opts$indexes_bandnames, custom_idx$indexes_bandnames), 
       indexes_formula = c(prod_opts$indexes_formula, custom_idx$indexes_formulas), 
       indexes_nodata_out = c(prod_opts$indexes_nodata_out, custom_idx$indexes_nodata_out), 
       quality_bandnames = prod_opts$quality_bandnames, quality_bandsel = general_opts$quality_bandsel, 
       quality_bitN = prod_opts$quality_bitN, quality_source = prod_opts$quality_source, 
       quality_nodata_in = prod_opts$quality_nodata_in, quality_nodata_out = prod_opts$quality_nodata_out, 
       file_prefixes = prod_opts$file_prefix, main_out_folder = prod_opts$main_out_folder, 
       gui = gui, use_aria = general_opts$use_aria, download_range = general_opts$download_range, 
       n_retries = n_retries, verbose = verbose) at /home/jeff/GitHub/MODIStsp/R/MODIStsp.R:464
3: MODIStsp_download(modislist, out_folder_mod, download_server, http, ftp, n_retries, 
       use_aria, date_dirs[date], year, DOY, user, password, sens_sel, date_name, gui, 
       mess_lab, verbose) at /home/jeff/GitHub/MODIStsp/R/MODIStsp_process.R:388
────────────────────────────────────────────────────────────────────────────────
✖ |  0 1     | MODIStsp Test 5: HTTP download from USGS, resize and reproject [14.5 s]
────────────────────────────────────────────────────────────────────────────────
test_MODIStsp.R:249: error: Tests on MODIStsp
argument is of length zero
1: MODIStsp(test = 5) at /home/jeff/GitHub/MODIStsp/tests/testthat/test_MODIStsp.R:249
2: MODIStsp_process(sel_prod = general_opts$sel_prod, start_date = general_opts$start_date, 
       end_date = general_opts$end_date, out_folder = general_opts$out_folder, out_folder_mod = general_opts$out_folder_mod, 
       reprocess = general_opts$reprocess, delete_hdf = general_opts$delete_hdf, sensor = general_opts$sensor, 
       download_server = general_opts$download_server, user = general_opts$user, password = general_opts$password, 
       https = prod_opts$http, ftps = prod_opts$ftp, start_x = general_opts$start_x, 
       start_y = general_opts$start_y, end_x = general_opts$end_x, end_y = general_opts$end_y, 
       full_ext = general_opts$full_ext, bbox = general_opts$bbox, out_format = general_opts$out_format, 
       out_res_sel = general_opts$out_res_sel, out_res = as.numeric(general_opts$out_res), 
       native_res = prod_opts$native_res, tiled = prod_opts$tiled, resampling = general_opts$resampling, 
       ts_format = general_opts$ts_format, compress = general_opts$compress, mod_proj_str = mod_proj_str, 
       outproj_str = general_opts$user_proj4, nodata_in = prod_opts$nodata_in, nodata_out = prod_opts$nodata_out, 
       rts = general_opts$rts, nodata_change = general_opts$nodata_change, scale_val = general_opts$scale_val, 
       scale_factor = prod_opts$scale_factor, offset = prod_opts$offset, datatype = prod_opts$datatype, 
       bandsel = general_opts$bandsel, bandnames = prod_opts$bandnames, indexes_bandsel = c(general_opts$indexes_bandsel), 
       indexes_bandnames = c(prod_opts$indexes_bandnames, custom_idx$indexes_bandnames), 
       indexes_formula = c(prod_opts$indexes_formula, custom_idx$indexes_formulas), 
       indexes_nodata_out = c(prod_opts$indexes_nodata_out, custom_idx$indexes_nodata_out), 
       quality_bandnames = prod_opts$quality_bandnames, quality_bandsel = general_opts$quality_bandsel, 
       quality_bitN = prod_opts$quality_bitN, quality_source = prod_opts$quality_source, 
       quality_nodata_in = prod_opts$quality_nodata_in, quality_nodata_out = prod_opts$quality_nodata_out, 
       file_prefixes = prod_opts$file_prefix, main_out_folder = prod_opts$main_out_folder, 
       gui = gui, use_aria = general_opts$use_aria, download_range = general_opts$download_range, 
       n_retries = n_retries, verbose = verbose) at /home/jeff/GitHub/MODIStsp/R/MODIStsp.R:464
3: MODIStsp_download(modislist, out_folder_mod, download_server, http, ftp, n_retries, 
       use_aria, date_dirs[date], year, DOY, user, password, sens_sel, date_name, gui, 
       mess_lab, verbose) at /home/jeff/GitHub/MODIStsp/R/MODIStsp_process.R:388
────────────────────────────────────────────────────────────────────────────────
  |======================================================================| 100%
  |======================================================================| 100%
  |======================================================================| 100%
  |======================================================================| 100%
✔ |  6       | MODIStsp Test 6: FTP download and mosaicing of MODIS tiles [77.0 s]
✔ |  1       | MODIStsp Test 7: Passing the extent with a spatial file [3.0 s]
⠏ |  0       | MODIStsp Test 8: Fail gracefully on missing connectionError : GET http://e4ftl01.cr.usgs.gov/MOTA/MCD43A3.006/
In addition: Warning message:
`encoding` is deprecated; all files now assumed to be UTF-8 
Error : GET ftp://ladsweb.nascom.nasa.gov/allData/6/MCD43A3/2017/
⠋ |  1       | MODIStsp Test 8: Fail gracefully on missing connectionError : GET ftp://ladsweb.nascom.nasa.gov/allData/6/MCD15A2H/2016/
✔ |  2       | MODIStsp Test 8: Fail gracefully on missing connection [0.5 s]
✖ |  0 1     | Check proper functioning of MODIStsp_readxml [1.9 s]
────────────────────────────────────────────────────────────────────────────────
test_MODIStsp_readxml.R:8: error: MODIStsp_readxml works as expected
attempt to select less than one element in OneIndex
1: MODIStsp_read_xml(tmpfile, system.file("ExtData", "MODIStsp_ProdOpts.xml", package = "MODIStsp")) at /home/jeff/GitHub/MODIStsp/tests/testthat/test_MODIStsp_readxml.R:8
────────────────────────────────────────────────────────────────────────────────
✔ |  2       | MODIStsp_resetindexes

══ Results ═════════════════════════════════════════════════════════════════════
Duration: 213.2 s

OK:       72
Failed:   3
Warnings: 2
Skipped:  0

Session information

R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] httptest_3.0.0       gWidgetsRGtk2_0.0-85 cairoDevice_2.24    
[4] gWidgets_0.0-54      RGtk2_2.20.34        MODIStsp_1.3.3.9000 
[7] testthat_2.0.0       devtools_1.13.4     

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.15        compiler_3.4.3      R.methodsS3_1.7.1  
 [4] bitops_1.0-6        R.utils_2.6.0       iterators_1.0.9    
 [7] tools_3.4.3         xts_0.10-1          gdalUtils_2.0.1.7  
[10] digest_0.6.15       jsonlite_1.5        memoise_1.1.0      
[13] lattice_0.20-35     rlang_0.1.6         foreach_1.4.4      
[16] cli_1.0.0           commonmark_1.4      curl_3.1           
[19] rgdal_1.2-16        parallel_3.4.3      withr_2.1.1        
[22] stringr_1.2.0       httr_1.3.1          roxygen2_6.0.1     
[25] raster_2.6-7        xml2_1.2.0          rgeos_0.3-26       
[28] grid_3.4.3          data.table_1.10.4-3 R6_2.2.2           
[31] pacman_0.4.6        sp_1.2-7            magrittr_1.5       
[34] codetools_0.2-15    assertthat_0.2.0    mime_0.5           
[37] stringi_1.1.6       RCurl_1.95-4.10     crayon_1.3.4       
[40] zoo_1.8-1           R.oo_1.21.0  

Let me know if there's any other info I can provide.

lbusett added a commit to ropensci/MODIStsp that referenced this issue Jan 31, 2018

Fix errors reported in ropensci review
See (ropensci/onboarding#184 (comment))

- FIx bug related to new xml version 
- Fix bug related to changes in metadata retreived from lpdaac
- bumps required xml2 version to 1.2.0
@lbusett

This comment has been minimized.

Contributor

lbusett commented Jan 31, 2018

@jeffreyhanson

Thanks! Glad you like our package!

Concerning the tests, thanks for reporting the issues. They should now be fixed. One was due to a recent change in xml2 functionality, the other (apparently) to a change in the "standard" http response of NASA servers. If you pull the new github version, the tests should pass (and you should also be able to download data from http also outside of the tests, which I think ws not possible due to the NASA response change. I'll have to also test the CRAN version to see if it still works over http.

@lwasser

This comment has been minimized.

lwasser commented Feb 5, 2018

hey @lbusett or @jeffreyhanson i'm a bit confused on the quartz install
screen shot 2018-02-05 at 11 33 57 am

i'm here:

next, edit the configure options for GTK to require x11 rather than Quartz:

brew edit gtk+

in the def install section, remove the reference to quartz and switch to:

--with-gdktarget=x11,
--enable-x11-backend

And would like to understand what part of the gtk+ file i need to edit. do i just replace "with-gdktarget=quartz", with two lines as follows:

"--with-gdktarget=x11",
"--enable-x11-backend",

it's taking me some time to get everything installed and this step is a bit confusing. thank you for any advice!

So i'm not sure if i did this right but another question. Does this message make sense:

library(RGtk2)
R session is headless; GTK+ not initialized.
@lbusett

This comment has been minimized.

Contributor

lbusett commented Feb 5, 2018

hi @lwasser ,

sorry you are having trouble... unfortunately, at the moent RGtk2 installation on mac is troublesome (and the solution to the trouble seems to change with time.. ;-(.
Moreover, I know almost nothing about mac, so I admit that I never really tested the instructions provided in the help.... However, those instructions mean that indeed you need to modify the gtk file so that it reads:

def install
args = ["--disable-dependency-tracking",
"--disable-silent-rules",
"--prefix=#{prefix}",
"--disable-glibtest",
"--enable-introspection=yes",
"--with-gdktarget=x11",
"--enable-x11-backend"
]

and then proceed with brew install.

Concerning the "R session is headless; GTK+ not initialized." : this does not bode well. It (should) mean that RGtk2 does not find a "graphical environment" to use. I fear that if you try to load gWidgetsRGtk2 you will have problems.

Again, sorry for the trouble. I hope you'll be able to solve it easily (this link may also help : https://gist.github.com/sebkopf/9405675). Otherwise, I'll try to get my hands on a (modern) mac pc and see if I manage to provide better instructions.

@lwasser

This comment has been minimized.

lwasser commented Feb 5, 2018

ok so remove "--disable-visibility" as well?
Yes i'm having many issues now. i can't seem to get cairoDevice or gWidgetsRGtk2 to install correctly. i'm not sure what to try next to get them to install.

> install.packages("cairoDevice", lib="~/Library/Frameworks/R.framework/Versions/3.4/Resources/library/cairoDevice")
Warning in install.packages :
  'lib = "~/Library/Frameworks/R.framework/Versions/3.4/Resources/library/cairoDevice"' is not writable
Would you like to use a personal library instead?  (y/n) y
Package which is only available in source form, and may need compilation of
  C/C++/Fortran: ‘cairoDevice’
Do you want to attempt to install these from sources?
y/n: y
installing the source package ‘cairoDevice’

trying URL 'https://cran.rstudio.com/src/contrib/cairoDevice_2.24.tar.gz'
Content type 'application/x-gzip' length 39639 bytes (38 KB)
==================================================
downloaded 38 KB

* installing *source* package ‘cairoDevice’ ...
** package ‘cairoDevice’ successfully unpacked and MD5 sums checked
checking for pkg-config... pkg-config
ERROR: gtk+2.0 not found by pkg-config.
ERROR: configuration failed for package ‘cairoDevice’
* removing ‘/Users/lewa8222/Library/R/3.4/library/cairoDevice’
Warning in install.packages :
  installation of package ‘cairoDevice’ had non-zero exit status

The downloaded source packages are in
	‘/private/var/folders/43/4q82487d5xsfpxdx6nl_c1wmhckx08/T/RtmpJoK8cV/downloaded_packages’

does this help at all with troubleshooting? if i say "n" to install from source it just quits. i'm wondering if i get this error because i edited the install code above incorrectly (i left disable-visibility in the code).

@lbusett

This comment has been minimized.

Contributor

lbusett commented Feb 5, 2018

Just a check:

did you already do

export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig:/usr/local/lib/pkgconfig/gtk+-2.0.pc:/opt/X11/lib/pkgconfig

which I think is setting the path to gtk+ and should solve to cairo installation issue (ERROR: gtk+2.0 not found by pkg-config.)?

@lwasser

This comment has been minimized.

lwasser commented Feb 5, 2018

ok!! i think i just got it. THANK YOU @lbusett i just installed those packages manually but also i hadn't run that line above that you suggested so i did that too. phew! i'll play with the package more tomorrow. thank you for the help.

@jeffreyhanson

This comment has been minimized.

jeffreyhanson commented Feb 7, 2018

Sorry for the late reply @lbusett, I just installed the latest GitHub version and all the tests pass now. Thank for you fixing this so quickly.

@lbusett

This comment has been minimized.

Contributor

lbusett commented Feb 7, 2018

Good. In the end, both problems were due to a breaking change in xml2. Since the cran version uses xml, I was not notified about it.

By the way: a user signalled us a bug in the gui that I probably introduced while refactoring the code for ropensci submission. I am going to commit a fix for it ASAP.

@lbusett

This comment has been minimized.

Contributor

lbusett commented Mar 12, 2018

Hi all,

at last, we managed to wrap-up an improved MODIStsp version addressing reviewers' suggestion. The current version can be installed with:

devtools::install_github("lbusett/MODIStsp")

In short, without going in detail through reviewers comments as in the last posts), here is a quick list of the implemented changes:

for @jeffreyhanson review:

  • All examples in vignettes and documentation now save by default in tempdir (also, at first GUI execution the output folders are set to $tempdir). If $tempdir$ is found as folder selection the outputs go to tempdir/MODStsp and tempdir/MODIStsp/HDFs

  • To avoid problems with new CRAN rules, at first MODIStsp execution the user is now asked permission to save MODIStsp previous options file in /Extdata/Previous

  • Rcurl dependency was removed. We now use httr::HEAD to check file size over ftp

  • We switched from "print" to "message" in MODIStsp_extract

  • We fixed cran warnings about "C:\Progra" etc.

  • We fixed the references in the PDF vignette

for @lwasser review:

  • We improved installation instructions for mac - fro both MODistsp package and GDAL

Improved instructions can be found at http://lbusett.github.io/MODIStsp/articles/installation.html#installing-on-mac and http://lbusett.github.io/MODIStsp/articles/installation.html#macos

Instructions were mainly derived from https://zhiyzuo.github.io/installation-rattle/. @lwasser : May you have a look and see if you think these are "good enough"?

GUI Improvements

We made various improvements in the GUI to address your comments (You can see a screenshot of the current GUI in my last post)

  • Improved the Product / Layers selection interface
  • The Product Information button is now also available in the main GUI
  • Selected layers are immediately visible in the Main Window. Help buttons were added to explain what the Quality Indicators and Spectral Indices are.
  • Added help suggestions on the "Add new index" functionality.

  • Strongly improved the "spatial extent" selection interface

  • Now it is possible to interactively select the MODIS tiles or custom spatial extent using the "Select from Map" button, and see the currently selected extent in a browser using the "Show Current Extent" button.
  • Improved / streamlined the "reprojection selection" section
  • Now only "Native" and "User Defined" can be chosen in the dropdown menu (befor, we also had some "random" projections in the dropdown because we use them often, but I think that this is cleaner)
  • When specifying a custom projection, in the Input field it is now possible to specify a projection simply by specifying an EPSG code or a UTM codes, instead of writing the full proj4 string;
  • Warning messages are now sent in case a User Projection is selected, to warn about possible problems introduced by reprojection;
  • Changing projection now re-sets the custom extent to avoid problems due to sequential reprojections
    of the extent (i.e., continuously enlarging bbox to avoid "losing" area).
  • Moved the hdf folder selector below the main output folder one and added help buttons on both output folder selectors to help the user

  • Modified the interface for selection of "formats" of virtual time series files.

  • Now we use a checkbox group allowing specifying which "time series formats" are desired, among R rasterStack, ENVI META or GDAL vrts (i.e., ENVI, R and GDAL timeseries are now "treated equally" and are selected using a single menu). We also added help buttons to facilitate the user.
  • Improved in general the help buttons and added some text formatting.

  • Changed the default output format to "GeoTiff", the default virtual raster files format to "rasterStack" and the default compression to "None"

These changes address most of the points raised by @lwasser , and I think that they in general improved user-friendliness of the interface (so, thanks for the suggestions!). The only point we were not able to address were those regarding:

When I have several boxes open (the select processing layers box and the insert new spectral index box) PLUS the original tsp interface... I have to figure out what order they are in so I can close them. Some users may think the entire thing is locked when really it’s just too many popups. This is a bit clunky for me to have 3 boxes and have to figure out the proper order! And it locked my computer up a bunch. I’m sure this is quartz specific but any way to control it?

Unfortunately, I see no easy way to do this because the different windows are "modal". I do not believe however this is a major issue. I can reevaluate if this is considered mandatory.

Other changes:

  • We improved functionality for dealing with NoData for products with multiple fill-values and fixed a couple of bugs affecting the "Seasonal" time series download (see ropensci/MODIStsp#113)

  • We made some changes in documentation. In particular, we improved some of the pkgdown articles to better help the users (see for example http://lbusett.github.io/MODIStsp/articles/noninteractive_execution.html and file:///D:/Documents/Source/git/MODIStsp/docs/articles/output.html)

  • To allow the new "interactive" spatial extent selection, we added "sf", "leaflet", "shiny", "mapview" and "mapedit" as "Suggests" in the DESCRIPTION file. If the user clicks on "Select from Map" and mapedit is not installed, he is prompted to install it. In this way, we avoid bringing in "heavy" dependencies. @karthik: do you think that is ok, or would it be better to put them in "Imports"?

OK, that's it. We hope you will like the new version and that we addressed all points raised in the review. Please let us know if you see any other issues or you need further changes/clarifications.

@lwasser

This comment has been minimized.

lwasser commented Mar 14, 2018

this all looks great @lbusett :) excited to see this package available on ropensci . it's a great contribution!!

@lbusett

This comment has been minimized.

Contributor

lbusett commented Mar 24, 2018

Hi @jeffreyhanson and @karthik ,

just wanted to ask if by any chance you had time to have a look at the implemented changes.

Asking because I am planning a CRAN release to introduce the new functionality (and - more importantly - the associated bug fixes...), but I'd rather do that at the end of the review process if possible.

@karthik

This comment has been minimized.

Member

karthik commented Mar 28, 2018

Hi @lbusett Will do shortly.

@lbusett

This comment has been minimized.

Contributor

lbusett commented Mar 28, 2018

@karthik Ok, thanks for the update!

@karthik

This comment has been minimized.

Member

karthik commented Apr 16, 2018

@jeffreyhanson Quick ping/reminder to sign off or ask more questions for @lbusett

🙏

@jeffreyhanson

This comment has been minimized.

jeffreyhanson commented Apr 18, 2018

I'm sorry for the late reply, this completely fell off my radar. @lbusett has done an amazing job and addressed all my issues/suggestions.

@lbusett

This comment has been minimized.

Contributor

lbusett commented Apr 21, 2018

@jeffreyhanson @lwasser Thanks! Glad you liked the improvements! I think the review process allowed to really improve the package!

@karthik How should we proceed from here? Consider that unfortunately i will be forced to make some changes with respect to the reviewed package due to some very revcent changes in NASA policies (e.g., discontinued support for FTP download, and discontinuation of most of version 005 products). I expect to be able to release a new version by the beginning of next week.

@karthik

This comment has been minimized.

Member

karthik commented Apr 25, 2018

@lbusett 👋 I'm glad to see @jeffreyhanson and @lwasser have signed off. I suggest making those changes and pushing a new release before we proceed with acceptance. I'm painfully aware of such policy changes.

@lbusett

This comment has been minimized.

Contributor

lbusett commented May 25, 2018

@karthik

I finally managed to put out a new version (1.3.4), including changes required to address changes in NASA policies. The new version is available on github, and I plan to upload it to CRAN soon.

I'd like therefore to ask:

  1. What are the next steps for approval?
  2. Can I proceed with CRAN release or do you prefer I wait for the end of the review/approval process?

Lorenzo

@lbusett

This comment has been minimized.

Contributor

lbusett commented Jun 11, 2018

@karthik, @jeffreyhanson, @lwasser

Hi all, please note that MODIStsp v 1.3.4 is now on cran. Please let me know what would be the next steps to be done for final approval on ropensci.

Thanks in advance!

Lorenzo

@lwasser

This comment has been minimized.

lwasser commented Jun 13, 2018

woo hoo! great work @lbusett ... seems like it's ready for final acceptance into ropensci by @karthik ?

@lbusett

This comment has been minimized.

Contributor

lbusett commented Jul 2, 2018

Hi,

just a quick ping for @karthik, since I saw you are back to work: How can we further proceed on this?

Lorenzo

@lbusett

This comment has been minimized.

Contributor

lbusett commented Jul 24, 2018

Hi @karthik

Sorry for bringing this up again, but it seems this submission got stuck in some kind of limbo, recently, and i would like to close this task. ☺️

Could we have any updates regarding next steps?

@lwasser

This comment has been minimized.

lwasser commented Aug 2, 2018

... i was just talking to someone about it and looking for this on ropensci and didn't see it and was wondering the same thing? @maelle @karthik just curious.

@sckott

This comment has been minimized.

Member

sckott commented Aug 2, 2018

Sorry about the delay, i'll ping Karthik

@karthik

This comment has been minimized.

Member

karthik commented Aug 3, 2018

@lbusett Apologies for the delay -- will follow up in a few.

@karthik

This comment has been minimized.

Member

karthik commented Aug 3, 2018

@lbusett Sincere apologies for the delay. Your last notification came during my holiday and got lost in a giant pile of unread GitHub notifications 😢But now that I see all the Is have been dotted and T's crossed, we are ready to go.

Congrats @lbusett, your submission has been approved! 🎉 Thank you for submitting and @lwasser and @jeffreyhanson for thorough and timely reviews. To-dos:

  • Transfer the repo to the rOpenSci organization under "Settings" in your repo. I have invited you to a team that should allow you to do so. You'll be made admin once you do.

  • Add the rOpenSci footer to the bottom of your README

[![ropensci_footer](https://ropensci.org/public_images/ropensci_footer.png)](https://ropensci.org)

  • Fix any links in badges for CI and coverage to point to the ropensci URL. (We'll turn on the services on our end as needed)

Welcome aboard! We'd also love a blog post about your package, either a short-form intro to it (https://ropensci.org/technotes/) or long-form post with more narrative about its development. ((https://ropensci.org/blog/). If you are, @stefaniebutland will be in touch about content and timing.

@karthik

This comment has been minimized.

Member

karthik commented Aug 3, 2018

@lbusett You can also now add the status badge (I should have alerted you earlier) to your README. It should switch to accepted shortly:

[![](https://badges.ropensci.org/184_status.svg)](https://github.com/ropensci/onboarding/issues/184)
@lbusett

This comment has been minimized.

Contributor

lbusett commented Aug 3, 2018

@karthik

thanks! No worries about the delay: I know it happens. I will do those last steps ASAP.
Concerning the blog post, although I already have some blog posts regarding the package here: https://lbusett.netlify.com/categories/modis/, I think that making also a technote on rOpensci would be nice.

@lwasser @jeffreyhanson

Thanks again for your very usefeul reviews: I think they allowed me to really improve the package!
I would like to include you as reviewers in the package DECRIPTION: would that be ok with you ?

Lorenzo

@stefaniebutland

This comment has been minimized.

stefaniebutland commented Aug 4, 2018

Hello @lbusett. Glad to hear you're interested in contributing a technote. It might get even more eyes on your work.

Examples of technotes are here: https://ropensci.org/technotes/
Here are some technical and editorial guidelines. https://github.com/ropensci/roweb2#contributing-a-blog-post. Difference for technote is categories: technotes in YAML

We ask that you submit your draft post via pull request a week before the planned publication date so we can give you some feedback.

Happy to answer any questions.

@lbusett

This comment has been minimized.

Contributor

lbusett commented Aug 7, 2018

Hi @karthik ,

I just transferred the repo to https://github.com/ropensci/MODIStsp after making the necessary changes.

(Note that the build on Travis is currently failing apparently due to a downtime of the NSIDC server)

@lwasser

This comment has been minimized.

lwasser commented Aug 7, 2018

@lbusett happy to be included as a reviewer in the description. thank you for your effort on this awesome package!! it's a great community contribution. i'm also happy to see it being embraced within the ropensci community!! yay!

@lbusett

This comment has been minimized.

Contributor

lbusett commented Aug 8, 2018

@lwasser Thanks! I will add you in the description ASAP!
@jeffreyhanson What about you? Isit ok also for you to be included as a reviewer ?

@lbusett

This comment has been minimized.

Contributor

lbusett commented Aug 8, 2018

@karthik

I am currently unable to modify the package settings (i.e., to modify the link to the website at the top of the repo page). Could you grant me admin settings on the repository ?

Lorenzo

@karthik karthik closed this Aug 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment