Skip to content

Commit

Permalink
Merge branch 'develop' into PecanProjectGH-2572-permission-issues-in-…
Browse files Browse the repository at this point in the history
…docker
  • Loading branch information
dlebauer committed Oct 14, 2020
2 parents d9b40e6 + eb55663 commit 154850b
Show file tree
Hide file tree
Showing 96 changed files with 2,954 additions and 4,056 deletions.
2 changes: 2 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,8 @@ jobs:
with:
key: install-${{ github.sha }}
path: .install
- name: update dependency lists
run: Rscript scripts/generate_dependencies.R
- name: build
run: make -j1
env:
Expand Down
2 changes: 2 additions & 0 deletions .github/workflows/styler-actions.yml
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ jobs:
uses: actions/download-artifact@v1
with:
name: artifacts
- name: update dependency lists
run: Rscript scripts/generate_dependencies.R
- name : make
shell: bash
run: |
Expand Down
6 changes: 5 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ For more information about this file see also [Keep a Changelog](http://keepacha
- Do not override meta-analysis settings random-effects = FALSE https://github.com/PecanProject/pecan/pull/2625
- model2netcdf.ED2 no longer detecting which varibles names `-T-` files have based on ED2 version (#2623)
- Changed docker-compose.yml to use user & group IDs of the operating system user (#2572)

-gSSURGO file download now added as inputs into BETY through extract_soil_gssurgo (#2666)

### Changed

Expand All @@ -38,15 +38,18 @@ For more information about this file see also [Keep a Changelog](http://keepacha
- Renamed functions that looked like S3 methods but were not:
* PEcAn.priors: `plot.posterior.density`->`plot_posterior.density`, `plot.prior.density`->`plot_prior.density`, `plot.trait`->`plot_trait` (#2439).
* PEcAn.visualization: `plot.netcdf`->`plot_netcdf` (#2526).
* PEcAn.assim.sequential: `Remote.Sync.launcher` -> `Remote_Sync_launcher` (#2652)
- Stricter package checking: `make check` and CI builds will now fail if `R CMD check` returns any ERRORs or any "newly-added" WARNINGs or NOTEs. "Newly-added" is determined by strict string comparison against a check result saved 2019-09-03; messages that exist in the reference result do not break the build but will be fixed as time allows in future refactorings (#2404).
- No longer writing an arbitrary num for each PFT, this was breaking ED runs potentially.
- The pecan/data container has no longer hardcoded path for postgres
- PEcAn.JULES: Removed dependency on `ncdf4.helpers` package, which has been removed from CRAN (#2511).
- data.remote: Arguments to the function `call_MODIS()` have been changed (issue #2519).
- Changed precipitaion downscale in `PEcAn.data.atmosphere::download.NOAA_GEFS_downscale`. Precipitation was being downscaled via a spline which was causing fake rain events. Instead the 6 hr precipitation flux values from GEFS are preserved with 0's filling in the hours between.
-Changed `dbfile.input.insert` to work with inputs (i.e soils) that don't have start and end dates associated with them

### Added

- New functionality to the PEcAn API to GET information about PFTs, formats & sites, submit workflows in XML or JSON formats & download relevant inputs/outputs/files related to runs & workflows (#2674 #2665 #2662 #2655)
- Functions to send/receive messages to/from rabbitmq.
- Documentation in [DEV-INTRO.md](DEV-INTRO.md) on development in a docker environment (#2553)
- PEcAn API that can be used to talk to PEcAn servers. Endpoints to GET the details about the server that user is talking to, PEcAn models, workflows & runs. Authetication enabled. (#2631)
Expand All @@ -63,6 +66,7 @@ For more information about this file see also [Keep a Changelog](http://keepacha

### Removed

- Removed the sugarcane and db folders from web, this removes the simple DB editor in the web folder. (#2532)
- Removed ED2IN.git (#2599) 'definitely going to break things for people' - but they can still use PEcAn <=1.7.1
- Database maintenance scripts `vacuum.bety.sh` and `reindex.bety.sh` have been moved to the [BeTY database repository](https://github.com/PecanProject/bety) (#2563).
- Scripts `dump.pgsql.sh` and `dump.mysql.sh` have been deleted. See the ["BeTY database administration"](https://pecanproject.github.io/pecan-documentation/develop/database.html) chapter of the PEcAn documentation for current recommendations (#2563).
Expand Down
2 changes: 1 addition & 1 deletion Makefile.depends
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ $(call depends,modules/assim.sequential): | .install/base/logger .install/base/r
$(call depends,modules/benchmark): | .install/modules/data.land .install/base/db .install/base/logger .install/base/remote .install/base/settings .install/base/utils
$(call depends,modules/data.atmosphere): | .install/base/db .install/base/logger .install/base/remote .install/base/utils
$(call depends,modules/data.hydrology): | .install/base/logger .install/base/utils
$(call depends,modules/data.land): | .install/base/db .install/base/utils .install/base/logger .install/base/remote .install/base/settings
$(call depends,modules/data.land): | .install/modules/benchmark .install/modules/data.atmosphere .install/base/db .install/base/logger .install/base/remote .install/base/settings .install/base/utils .install/base/visualization
$(call depends,modules/data.remote): | .install/base/db .install/base/utils .install/base/logger .install/base/remote
$(call depends,modules/emulator): | .install/base/logger
$(call depends,modules/meta.analysis): | .install/base/utils .install/base/db .install/base/logger .install/base/settings
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ PEcAn is not itself an ecosystem model, and it can be used to with a variety of

## Documentation

Consult our [Documentation](https://pecanproject.github.io/pecan-documentation/) for full documentation of the PEcAn Project.
Consult documentation of the PEcAn Project; either the [lastest stable development](https://pecanproject.github.io/pecan-documentation/develop/) branch, the latest [release](https://pecanproject.github.io/pecan-documentation/master/). Documentation from [earlier releases is here](https://pecanproject.github.io/documentation.html).

## Getting Started

Expand Down
8 changes: 6 additions & 2 deletions apps/api/R/auth.R
Original file line number Diff line number Diff line change
Expand Up @@ -51,12 +51,16 @@ validate_crypt_pass <- function(username, crypt_pass) {
#* @return Appropriate response
#* @author Tezan Sahu
authenticate_user <- function(req, res) {
# Fix CORS issues
res$setHeader("Access-Control-Allow-Origin", "*")

# If the API endpoint that do not require authentication
if (
Sys.getenv("AUTH_REQ") == FALSE ||
grepl("swagger", req$PATH_INFO, ignore.case = TRUE) ||
grepl("openapi.json", req$PATH_INFO, fixed = TRUE) ||
grepl("ping", req$PATH_INFO, ignore.case = TRUE) ||
grepl("status", req$PATH_INFO, ignore.case = TRUE))
grepl("/api/ping", req$PATH_INFO, ignore.case = TRUE) ||
grepl("/api/status", req$PATH_INFO, ignore.case = TRUE))
{
req$user$userid <- NA
req$user$username <- ""
Expand Down
8 changes: 8 additions & 0 deletions apps/api/R/entrypoint.R
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,14 @@ root$mount("/api/sites", sites_pr)
pfts_pr <- plumber::plumber$new("pfts.R")
root$mount("/api/pfts", pfts_pr)

# The endpoints mounted here are related to details of PEcAn formats
formats_pr <- plumber::plumber$new("formats.R")
root$mount("/api/formats", formats_pr)

# The endpoints mounted here are related to details of PEcAn inputs
inputs_pr <- plumber::plumber$new("inputs.R")
root$mount("/api/inputs", inputs_pr)

# The endpoints mounted here are related to details of PEcAn workflows
workflows_pr <- plumber::plumber$new("workflows.R")
root$mount("/api/workflows", workflows_pr)
Expand Down
89 changes: 89 additions & 0 deletions apps/api/R/formats.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
library(dplyr)

#' Retrieve the details of a PEcAn format, based on format_id
#' @param format_id Format ID (character)
#' @return Format details
#' @author Tezan Sahu
#* @get /<format_id>
getFormat <- function(format_id, res){

dbcon <- PEcAn.DB::betyConnect()

Format <- tbl(dbcon, "formats") %>%
select(format_id = id, name, notes, header, mimetype_id) %>%
filter(format_id == !!format_id)

Format <- tbl(dbcon, "mimetypes") %>%
select(mimetype_id = id, mimetype = type_string) %>%
inner_join(Format, by = "mimetype_id") %>%
select(-mimetype_id)

qry_res <- Format %>% collect()

if (nrow(qry_res) == 0) {
PEcAn.DB::db.close(dbcon)
res$status <- 404
return(list(error="Format not found"))
}
else {
# Convert the response from tibble to list
response <- list()
for(colname in colnames(qry_res)){
response[colname] <- qry_res[colname]
}

format_vars <- tbl(dbcon, "formats_variables") %>%
select(name, unit, format_id, variable_id) %>%
filter(format_id == !!format_id)
format_vars <- tbl(dbcon, "variables") %>%
select(variable_id = id, description, units) %>%
inner_join(format_vars, by="variable_id") %>%
mutate(unit = ifelse(unit %in% "", units, unit)) %>%
select(-variable_id, -format_id, -units) %>%
collect()

PEcAn.DB::db.close(dbcon)

response$format_variables <- format_vars
return(response)
}
}

#########################################################################

#' Search for PEcAn format(s) containing wildcards for filtering
#' @param format_name Format name search string (character)
#' @param mimetype Mime type search string (character)
#' @param ignore_case Logical. If `TRUE` (default) use case-insensitive search otherwise, use case-sensitive search
#' @return Formats subset matching the model search string
#' @author Tezan Sahu
#* @get /
searchFormats <- function(format_name="", mimetype="", ignore_case=TRUE, res){
format_name <- URLdecode(format_name)
mimetype <- URLdecode(mimetype)

dbcon <- PEcAn.DB::betyConnect()

Formats <- tbl(dbcon, "formats") %>%
select(format_id = id, format_name=name, mimetype_id) %>%
filter(grepl(!!format_name, format_name, ignore.case=ignore_case))

Formats <- tbl(dbcon, "mimetypes") %>%
select(mimetype_id = id, mimetype = type_string) %>%
inner_join(Formats, by = "mimetype_id") %>%
filter(grepl(!!mimetype, mimetype, ignore.case=ignore_case)) %>%
select(-mimetype_id) %>%
arrange(format_id)

qry_res <- Formats %>% collect()

PEcAn.DB::db.close(dbcon)

if (nrow(qry_res) == 0) {
res$status <- 404
return(list(error="Format(s) not found"))
}
else {
return(list(formats=qry_res, count = nrow(qry_res)))
}
}
1 change: 1 addition & 0 deletions apps/api/R/general.R
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ status <- function() {

dbcon <- PEcAn.DB::betyConnect()
res <- list(host_details = PEcAn.DB::dbHostInfo(dbcon))
res$host_details$authentication_required = get_env_var("AUTH_REQ")

res$pecan_details <- list(
version = get_env_var("PECAN_VERSION"),
Expand Down
38 changes: 38 additions & 0 deletions apps/api/R/get.file.R
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
library(dplyr)

get.file <- function(filepath, userid) {
# Check if the file path is valid
if(! file.exists(filepath)){
return(list(status = "Error", message = "File not found"))
}

# Check if the workflow for run after obtaining absolute path is owned by the user or not
parent_dir <- normalizePath(dirname(filepath))

run_id <- substr(parent_dir, stringi::stri_locate_last(parent_dir, regex="/")[1] + 1, stringr::str_length(parent_dir))

if(Sys.getenv("AUTH_REQ") == TRUE) {
dbcon <- PEcAn.DB::betyConnect()

Run <- tbl(dbcon, "runs") %>%
filter(id == !!run_id)
Run <- tbl(dbcon, "ensembles") %>%
select(ensemble_id=id, workflow_id) %>%
full_join(Run, by="ensemble_id") %>%
filter(id == !!run_id)
user_id <- tbl(dbcon, "workflows") %>%
select(workflow_id=id, user_id) %>% full_join(Run, by="workflow_id") %>%
filter(id == !!run_id) %>%
pull(user_id)

PEcAn.DB::db.close(dbcon)

if(! user_id == userid) {
return(list(status = "Error", message = "Access forbidden"))
}
}

# Read the data in binary form & return it
bin <- readBin(filepath,'raw', n = file.info(filepath)$size)
return(list(file_contents = bin))
}

0 comments on commit 154850b

Please sign in to comment.