diff --git a/.gitignore b/.gitignore index 227ebc1..39f6c58 100644 --- a/.gitignore +++ b/.gitignore @@ -6,4 +6,4 @@ hdar.Rproj inst/doc /Meta/ -/doc/ +doc/ diff --git a/DESCRIPTION b/DESCRIPTION index 71ab474..7a67407 100644 --- a/DESCRIPTION +++ b/DESCRIPTION @@ -1,19 +1,19 @@ Package: hdar Type: Package Title: 'REST' API Client for Accessing Data on 'WEkEO HDA V2' -Version: 1.0.0 +Version: 1.0.2 URL: https://www.wekeo.eu/ BugReports: https://github.com/eea/hdar/issues -Authors@R: person("Matteo", "Mattiuzzi", email = "matteo@mattiuzzi.com", role = c("aut", "cre")) -Maintainer: Matteo Mattiuzzi +Authors@R: person("Matteo", "Mattiuzzi", email = "matteo.mattiuzzi@eea.europa.eu", role = c("aut", "cre")) +Maintainer: Matteo Mattiuzzi Description: - The `hdar` R package offers seamless access to the 'WEkEO Harmonised Data Access (HDA)' API, - enabling users to query, download, and process data efficiently from the HDA platform. - With `hdar`, researchers and data scientists can integrate HDA's extensive datasets - into their R workflows, enhancing their data analysis capabilities. The API documentation - is available at [https://gateway.prod.wekeo2.eu/hda-broker/docs]https://gateway.prod.wekeo2.eu/hda-broker/docs, providing comprehensive - information on the API's functionalities and usage. -License: EUPL (>= 1.2) + Provides seamless access to the WEkEO Harmonised Data Access (HDA) API, enabling + users to query, download, and process data efficiently from the HDA platform. + With 'hdar', researchers and data scientists can integrate the extensive HDA + datasets into their R workflows, enhancing their data analysis capabilities. + Comprehensive information on the API functionality and usage is available + at . +License: EUPL (>= 1.2) Encoding: UTF-8 RoxygenNote: 7.3.2 Imports: diff --git a/LICENSE b/LICENSE deleted file mode 100644 index 4153cd3..0000000 --- a/LICENSE +++ /dev/null @@ -1,287 +0,0 @@ - EUROPEAN UNION PUBLIC LICENCE v. 1.2 - EUPL © the European Union 2007, 2016 - -This European Union Public Licence (the ‘EUPL’) applies to the Work (as defined -below) which is provided under the terms of this Licence. Any use of the Work, -other than as authorised under this Licence is prohibited (to the extent such -use is covered by a right of the copyright holder of the Work). - -The Work is provided under the terms of this Licence when the Licensor (as -defined below) has placed the following notice immediately following the -copyright notice for the Work: - - Licensed under the EUPL - -or has expressed by any other means his willingness to license under the EUPL. - -1. Definitions - -In this Licence, the following terms have the following meaning: - -- ‘The Licence’: this Licence. - -- ‘The Original Work’: the work or software distributed or communicated by the - Licensor under this Licence, available as Source Code and also as Executable - Code as the case may be. - -- ‘Derivative Works’: the works or software that could be created by the - Licensee, based upon the Original Work or modifications thereof. This Licence - does not define the extent of modification or dependence on the Original Work - required in order to classify a work as a Derivative Work; this extent is - determined by copyright law applicable in the country mentioned in Article 15. - -- ‘The Work’: the Original Work or its Derivative Works. - -- ‘The Source Code’: the human-readable form of the Work which is the most - convenient for people to study and modify. - -- ‘The Executable Code’: any code which has generally been compiled and which is - meant to be interpreted by a computer as a program. - -- ‘The Licensor’: the natural or legal person that distributes or communicates - the Work under the Licence. - -- ‘Contributor(s)’: any natural or legal person who modifies the Work under the - Licence, or otherwise contributes to the creation of a Derivative Work. - -- ‘The Licensee’ or ‘You’: any natural or legal person who makes any usage of - the Work under the terms of the Licence. - -- ‘Distribution’ or ‘Communication’: any act of selling, giving, lending, - renting, distributing, communicating, transmitting, or otherwise making - available, online or offline, copies of the Work or providing access to its - essential functionalities at the disposal of any other natural or legal - person. - -2. Scope of the rights granted by the Licence - -The Licensor hereby grants You a worldwide, royalty-free, non-exclusive, -sublicensable licence to do the following, for the duration of copyright vested -in the Original Work: - -- use the Work in any circumstance and for all usage, -- reproduce the Work, -- modify the Work, and make Derivative Works based upon the Work, -- communicate to the public, including the right to make available or display - the Work or copies thereof to the public and perform publicly, as the case may - be, the Work, -- distribute the Work or copies thereof, -- lend and rent the Work or copies thereof, -- sublicense rights in the Work or copies thereof. - -Those rights can be exercised on any media, supports and formats, whether now -known or later invented, as far as the applicable law permits so. - -In the countries where moral rights apply, the Licensor waives his right to -exercise his moral right to the extent allowed by law in order to make effective -the licence of the economic rights here above listed. - -The Licensor grants to the Licensee royalty-free, non-exclusive usage rights to -any patents held by the Licensor, to the extent necessary to make use of the -rights granted on the Work under this Licence. - -3. Communication of the Source Code - -The Licensor may provide the Work either in its Source Code form, or as -Executable Code. If the Work is provided as Executable Code, the Licensor -provides in addition a machine-readable copy of the Source Code of the Work -along with each copy of the Work that the Licensor distributes or indicates, in -a notice following the copyright notice attached to the Work, a repository where -the Source Code is easily and freely accessible for as long as the Licensor -continues to distribute or communicate the Work. - -4. Limitations on copyright - -Nothing in this Licence is intended to deprive the Licensee of the benefits from -any exception or limitation to the exclusive rights of the rights owners in the -Work, of the exhaustion of those rights or of other applicable limitations -thereto. - -5. Obligations of the Licensee - -The grant of the rights mentioned above is subject to some restrictions and -obligations imposed on the Licensee. Those obligations are the following: - -Attribution right: The Licensee shall keep intact all copyright, patent or -trademarks notices and all notices that refer to the Licence and to the -disclaimer of warranties. The Licensee must include a copy of such notices and a -copy of the Licence with every copy of the Work he/she distributes or -communicates. The Licensee must cause any Derivative Work to carry prominent -notices stating that the Work has been modified and the date of modification. - -Copyleft clause: If the Licensee distributes or communicates copies of the -Original Works or Derivative Works, this Distribution or Communication will be -done under the terms of this Licence or of a later version of this Licence -unless the Original Work is expressly distributed only under this version of the -Licence — for example by communicating ‘EUPL v. 1.2 only’. The Licensee -(becoming Licensor) cannot offer or impose any additional terms or conditions on -the Work or Derivative Work that alter or restrict the terms of the Licence. - -Compatibility clause: If the Licensee Distributes or Communicates Derivative -Works or copies thereof based upon both the Work and another work licensed under -a Compatible Licence, this Distribution or Communication can be done under the -terms of this Compatible Licence. For the sake of this clause, ‘Compatible -Licence’ refers to the licences listed in the appendix attached to this Licence. -Should the Licensee's obligations under the Compatible Licence conflict with -his/her obligations under this Licence, the obligations of the Compatible -Licence shall prevail. - -Provision of Source Code: When distributing or communicating copies of the Work, -the Licensee will provide a machine-readable copy of the Source Code or indicate -a repository where this Source will be easily and freely available for as long -as the Licensee continues to distribute or communicate the Work. - -Legal Protection: This Licence does not grant permission to use the trade names, -trademarks, service marks, or names of the Licensor, except as required for -reasonable and customary use in describing the origin of the Work and -reproducing the content of the copyright notice. - -6. Chain of Authorship - -The original Licensor warrants that the copyright in the Original Work granted -hereunder is owned by him/her or licensed to him/her and that he/she has the -power and authority to grant the Licence. - -Each Contributor warrants that the copyright in the modifications he/she brings -to the Work are owned by him/her or licensed to him/her and that he/she has the -power and authority to grant the Licence. - -Each time You accept the Licence, the original Licensor and subsequent -Contributors grant You a licence to their contributions to the Work, under the -terms of this Licence. - -7. Disclaimer of Warranty - -The Work is a work in progress, which is continuously improved by numerous -Contributors. It is not a finished work and may therefore contain defects or -‘bugs’ inherent to this type of development. - -For the above reason, the Work is provided under the Licence on an ‘as is’ basis -and without warranties of any kind concerning the Work, including without -limitation merchantability, fitness for a particular purpose, absence of defects -or errors, accuracy, non-infringement of intellectual property rights other than -copyright as stated in Article 6 of this Licence. - -This disclaimer of warranty is an essential part of the Licence and a condition -for the grant of any rights to the Work. - -8. Disclaimer of Liability - -Except in the cases of wilful misconduct or damages directly caused to natural -persons, the Licensor will in no event be liable for any direct or indirect, -material or moral, damages of any kind, arising out of the Licence or of the use -of the Work, including without limitation, damages for loss of goodwill, work -stoppage, computer failure or malfunction, loss of data or any commercial -damage, even if the Licensor has been advised of the possibility of such damage. -However, the Licensor will be liable under statutory product liability laws as -far such laws apply to the Work. - -9. Additional agreements - -While distributing the Work, You may choose to conclude an additional agreement, -defining obligations or services consistent with this Licence. However, if -accepting obligations, You may act only on your own behalf and on your sole -responsibility, not on behalf of the original Licensor or any other Contributor, -and only if You agree to indemnify, defend, and hold each Contributor harmless -for any liability incurred by, or claims asserted against such Contributor by -the fact You have accepted any warranty or additional liability. - -10. Acceptance of the Licence - -The provisions of this Licence can be accepted by clicking on an icon ‘I agree’ -placed under the bottom of a window displaying the text of this Licence or by -affirming consent in any other similar way, in accordance with the rules of -applicable law. Clicking on that icon indicates your clear and irrevocable -acceptance of this Licence and all of its terms and conditions. - -Similarly, you irrevocably accept this Licence and all of its terms and -conditions by exercising any rights granted to You by Article 2 of this Licence, -such as the use of the Work, the creation by You of a Derivative Work or the -Distribution or Communication by You of the Work or copies thereof. - -11. Information to the public - -In case of any Distribution or Communication of the Work by means of electronic -communication by You (for example, by offering to download the Work from a -remote location) the distribution channel or media (for example, a website) must -at least provide to the public the information requested by the applicable law -regarding the Licensor, the Licence and the way it may be accessible, concluded, -stored and reproduced by the Licensee. - -12. Termination of the Licence - -The Licence and the rights granted hereunder will terminate automatically upon -any breach by the Licensee of the terms of the Licence. - -Such a termination will not terminate the licences of any person who has -received the Work from the Licensee under the Licence, provided such persons -remain in full compliance with the Licence. - -13. Miscellaneous - -Without prejudice of Article 9 above, the Licence represents the complete -agreement between the Parties as to the Work. - -If any provision of the Licence is invalid or unenforceable under applicable -law, this will not affect the validity or enforceability of the Licence as a -whole. Such provision will be construed or reformed so as necessary to make it -valid and enforceable. - -The European Commission may publish other linguistic versions or new versions of -this Licence or updated versions of the Appendix, so far this is required and -reasonable, without reducing the scope of the rights granted by the Licence. New -versions of the Licence will be published with a unique version number. - -All linguistic versions of this Licence, approved by the European Commission, -have identical value. Parties can take advantage of the linguistic version of -their choice. - -14. Jurisdiction - -Without prejudice to specific agreement between parties, - -- any litigation resulting from the interpretation of this License, arising - between the European Union institutions, bodies, offices or agencies, as a - Licensor, and any Licensee, will be subject to the jurisdiction of the Court - of Justice of the European Union, as laid down in article 272 of the Treaty on - the Functioning of the European Union, - -- any litigation arising between other parties and resulting from the - interpretation of this License, will be subject to the exclusive jurisdiction - of the competent court where the Licensor resides or conducts its primary - business. - -15. Applicable Law - -Without prejudice to specific agreement between parties, - -- this Licence shall be governed by the law of the European Union Member State - where the Licensor has his seat, resides or has his registered office, - -- this licence shall be governed by Belgian law if the Licensor has no seat, - residence or registered office inside a European Union Member State. - -Appendix - -‘Compatible Licences’ according to Article 5 EUPL are: - -- GNU General Public License (GPL) v. 2, v. 3 -- GNU Affero General Public License (AGPL) v. 3 -- Open Software License (OSL) v. 2.1, v. 3.0 -- Eclipse Public License (EPL) v. 1.0 -- CeCILL v. 2.0, v. 2.1 -- Mozilla Public Licence (MPL) v. 2 -- GNU Lesser General Public Licence (LGPL) v. 2.1, v. 3 -- Creative Commons Attribution-ShareAlike v. 3.0 Unported (CC BY-SA 3.0) for - works other than software -- European Union Public Licence (EUPL) v. 1.1, v. 1.2 -- Québec Free and Open-Source Licence — Reciprocity (LiLiQ-R) or Strong - Reciprocity (LiLiQ-R+). - -The European Commission may update this Appendix to later versions of the above -licences without producing a new version of the EUPL, as long as they provide -the rights granted in Article 2 of this Licence and protect the covered Source -Code from exclusive appropriation. - -All other changes or additions to this Appendix require the production of a new -EUPL version. diff --git a/NAMESPACE b/NAMESPACE index b4ad2d8..221df14 100644 --- a/NAMESPACE +++ b/NAMESPACE @@ -13,6 +13,7 @@ importFrom(httr2,req_body_json) importFrom(httr2,req_method) importFrom(httr2,req_url_query) importFrom(httr2,request) +importFrom(humanize,natural_size) importFrom(jsonlite,toJSON) importFrom(magrittr,"%>%") importFrom(stringr,str_detect) diff --git a/R/client.R b/R/client.R index e50671b..d32a34f 100644 --- a/R/client.R +++ b/R/client.R @@ -22,8 +22,10 @@ Client <- R6::R6Class("Client", #' @return An instance of the `Client` class. #' @export initialize = function(user, password, save_credentials = FALSE) { + private$credentials_file_path <- "~/.hdarc" + if (missing(user) || missing(password)) { - # read from ~/.hdrc file + # read from file cred <- private$read_credentials_from_file() user <- cred[1] password <- cred[2] @@ -72,7 +74,6 @@ Client <- R6::R6Class("Client", private$auth$get_token() } - req <- req %>% httr2::req_headers(Authorization = paste("Bearer", private$auth$token())) %>% httr2::req_retry(max_tries = 3) @@ -82,7 +83,9 @@ Client <- R6::R6Class("Client", req %>% httr2::req_perform() }, error = function(err) { - if (httr2::last_response()$status_code == 403 || httr2::last_response()$status_code == 401) { + resp <- httr2::last_response() + + if (resp$status_code == 403 || resp$status_code == 401) { private$auth$get_token() req <- req %>% httr2::req_headers(Authorization = paste("Bearer", private$auth$token())) @@ -91,11 +94,13 @@ Client <- R6::R6Class("Client", req %>% httr2::req_perform() }, error = function(err) { - stop(paste("Network error. Reason: ", err)) + error_message <- private$extract_error_message(resp) + stop(paste("Network Error:", error_message, sep = "\n")) } ) } else { - stop(paste("Network error. Reason: ", err)) + error_message <- private$extract_error_message(resp) + stop(paste("Network Error:", error_message, sep = "\n")) } } ) @@ -114,7 +119,7 @@ Client <- R6::R6Class("Client", } return(list("data" = data, "status_code" = resp$status_code)) } - stop(httr2::resp_body_json(resp)$detail) + stop(paste("Incorrect data: ", httr2::resp_body_json(resp)$detail, sep = "\n")) }, @@ -276,21 +281,22 @@ Client <- R6::R6Class("Client", #' @description #' This function performs a search based on a specified query and returns an instance of \code{\link{SearchResults}}. #' - #' @param query Character string representing the search query. + #' @param json_query Character string representing the search query. #' @param limit Optional; a number specifying the maximum number of results to return. #' @return An instance of the \code{\link{SearchResults}} class containing the search results. #' @seealso \code{\link[=SearchResults]{SearchResults}} for details on the returned object. #' @importFrom httr2 request req_method req_body_json #' @importFrom stringr str_detect + #' @importFrom humanize natural_size #' @export - search = function(query, limit = NULL) { - json_query <- jsonlite::toJSON(query, pretty = TRUE, auto_unbox = FALSE) + search = function(json_query, limit = NULL) { json_query <- strip_off_template_placeholders(json_query) + query <- jsonlite::fromJSON(json_query, simplifyVector = FALSE) url <- paste0(self$apiUrl, "/dataaccess/search") req <- httr2::request(url) %>% httr2::req_method("POST") %>% - httr2::req_body_json(jsonlite::fromJSON(json_query)) + httr2::req_body_json(query) tryCatch( { @@ -305,7 +311,8 @@ Client <- R6::R6Class("Client", search_results }, error = function(err) { - stop(paste("Search query failed")) + warning("Search query failed") + stop(err) } ) }, @@ -334,7 +341,8 @@ Client <- R6::R6Class("Client", resp <- self$send_request(req)$data if (to_json) { - resp <- jsonlite::toJSON(resp, pretty = TRUE, auto_unbox = TRUE) + resp <- jsonlite::toJSON(resp, pretty = TRUE, auto_unbox = TRUE, digits = 17) + print(resp) } resp }, @@ -353,12 +361,13 @@ Client <- R6::R6Class("Client", ), private = list( auth = NULL, + credentials_file_path = NULL, read_credentials_from_file = function() { - if (!file.exists("~/.hdarc")) { + if (!file.exists(private$credentials_file_path)) { return(c("", "")) } - file <- readLines("~/.hdarc") + file <- readLines(private$credentials_file_path) user <- private$read_credential_property_from_file(file, "user") password <- private$read_credential_property_from_file(file, "password") @@ -374,11 +383,11 @@ Client <- R6::R6Class("Client", prop_value <- gsub(regexp, "\\1", file[idx]) %>% trimws() }, save_credentials_to_file = function(user, pwd) { - if (!file.exists("~/.hdarc")) { - file.create("~/.hdarc") + if (!file.exists(private$credentials_file_path)) { + file.create(private$credentials_file_path) } - fileConn <- file("~/.hdarc") + fileConn <- file(private$credentials_file_path) writeLines( c( paste0("user:", user), @@ -397,8 +406,14 @@ Client <- R6::R6Class("Client", for (param in names(data)) { if (param == "dataset_id") next - if (param == "itemsPerPage") next - if (param == "startIndex") next + if (param == "itemsPerPage") { + obj$itemsPerPage <- 11 + next + } + if (param == "startIndex") { + obj$startIndex <- 0 + next + } if (is.null(data[[param]])) next if (grepl("bbox", param, fixed = TRUE)) { @@ -412,22 +427,18 @@ Client <- R6::R6Class("Client", next } - pValue <- extract_template_param_default_value(data[[param]]) - if (is.null(pValue)) { - switch(param, - "itemsPerPage" = { - pValue <- 11 - }, - "startIndex" = { - pValue <- 0 - }, - next - ) + param_meta <- extract_param_metadata(data[[param]]) + obj <- c(obj, setNames(param_meta$value, param)) + if (!is.na(param_meta$comment)) { + obj <- c(obj, setNames(param_meta$comment, paste0("_comment_", param))) + } + if (!is.na(param_meta$possible_values)) { + obj <- c(obj, setNames(param_meta$possible_values, paste0("_values_", param))) } - obj <- c(obj, setNames(pValue, param)) } + if (to_json) { - jsonlite::toJSON(obj, pretty = TRUE, auto_unbox = TRUE) + jsonlite::toJSON(obj, pretty = TRUE, auto_unbox = TRUE, digits = 17) } else { obj } @@ -490,6 +501,18 @@ Client <- R6::R6Class("Client", "doi" = doi, "thumbnails" = meta[["thumbnails"]] ) + }, + extract_error_message = function(resp) { + content_type <- httr2::resp_content_type(resp) + + if (grepl("application/json", content_type)) { + resp %>% + httr2::resp_body_json() %>% + jsonlite::toJSON(pretty = TRUE, auto_unbox = TRUE, digits = 17) + } else { + # For other content types (e.g., text) + resp %>% httr2::resp_body_string() + } } ) ) diff --git a/R/paginator.R b/R/paginator.R index ae9f239..0fea034 100644 --- a/R/paginator.R +++ b/R/paginator.R @@ -26,7 +26,7 @@ Paginator <- R6::R6Class("Paginator", } } - if ((!is.null(limit) && length(results) >= limit) || + if ((!is.null(limit) && length(results) > limit) || length(results) >= resp$properties$totalResults || length(results) == 0) { break } @@ -37,7 +37,8 @@ Paginator <- R6::R6Class("Paginator", results }, error = function(err) { - stop(paste("Error when getting data. Reason: ", err)) + error = paste("Error when getting data with paginator", err, sep = "\n") + stop(error) } ) } diff --git a/R/search_results.R b/R/search_results.R index 8c314ef..a0ff289 100644 --- a/R/search_results.R +++ b/R/search_results.R @@ -33,7 +33,11 @@ SearchResults <- R6::R6Class("SearchResults", self$total_count <- length(sapply(results, FUN = function(x) { x$id })) - self$total_size <- sum(sapply(results, function(x) if (!is.null(x$properties$size)) x$properties$size else 0)) + if (self$total_count > 0) { + self$total_size <- sum(sapply(results, function(x) if (!is.null(x$properties$size) && is.numeric(x$properties$size)) x$properties$size else 0)) + } else { + self$total_size <- 0 + } }, #' @description @@ -41,9 +45,10 @@ SearchResults <- R6::R6Class("SearchResults", #' @param output_dir A string specifying the directory where downloaded files will be saved. #' @param selected_indexes Optional; indices of the specific results to download. #' @param stop_at_failure Optional; controls whether the download process of multiple files should immediately stop upon encountering the first failure. + #' @param force Optional; forces the download even if the file already exists in the specified output directory. #' @return Nothing returned but downloaded files are saved at the specified location. #' @export - download = function(output_dir, selected_indexes, stop_at_failure = TRUE) { + download = function(output_dir, selected_indexes, stop_at_failure = TRUE, force = FALSE) { if (self$total_count == 0 || !private$prompt_user_confirmation(self$total_size)) { return(NULL) } @@ -73,12 +78,11 @@ SearchResults <- R6::R6Class("SearchResults", download_id <- private$get_download_id(r) is_ready <- private$ensure_download_is_ready(download_id) if (is_ready) { - private$download_resource(download_id, output_dir) + private$download_resource(download_id, output_dir, force) } }, error = function(err) { - warning(err) - warning("[!] An error occurred during the download.") + warning("Error during the downloading:", err, sep = "\n") should_break <<- stop_at_failure } ) @@ -121,7 +125,7 @@ SearchResults <- R6::R6Class("SearchResults", resp <- private$client$send_request(req)$data resp$download_id }, - download_resource = function(download_id, output_dir) { + download_resource = function(download_id, output_dir, force = FALSE) { url <- paste0(private$client$apiUrl, "/dataaccess/download/", download_id) req <- httr2::request(url) %>% httr2::req_method("GET") @@ -140,8 +144,15 @@ SearchResults <- R6::R6Class("SearchResults", "downloaded_file" } - # Write the content to a file + # Define the full file path file_path <- file.path(output_dir, filename) + + # Check if the file already exists and force flag is FALSE + if (file.exists(file_path) && !force) { + warning("File already exists:", file_path, "- Skipping download.\n") + return(NA) + } + writeBin(httr2::resp_body_raw(resp), file_path) return(NA) @@ -149,12 +160,12 @@ SearchResults <- R6::R6Class("SearchResults", prompt_user_confirmation = function(total_size) { if (total_size >= private$LARGE_DOWNLOAD_SIZE) { repeat { - cat("The total size is", humanize::natural_size(total_size), ". Do you want to proceed? (Y/N): ") + message("The total size is ", humanize::natural_size(total_size), ". Do you want to proceed? (Y/N): ") answer <- tolower(readLines(n = 1)) if (answer %in% c("y", "n")) { return(answer == "y") } else { - cat("Invalid input. Please enter 'Y' or 'N'.\n") + message("Invalid input. Please enter 'Y' or 'N'.\n") return(TRUE) } } diff --git a/R/utils.R b/R/utils.R index fbc6812..5c98a24 100644 --- a/R/utils.R +++ b/R/utils.R @@ -3,66 +3,122 @@ is_valid_url <- function(url) { grepl(url_pattern, url) } -PLACEHOLDER_TAG = "__###" +PLACEHOLDER_TAG <- "__###" is_placeholder <- function(value) { is_single_string(value) && stringr::str_detect(value, paste0("^", PLACEHOLDER_TAG)) } -extract_template_param_string_default_value <- function(meta) { +extract_param_meta_for_string <- function(meta) { value <- NULL + comment <- NA + possible_values <- NA if (exists("default", where = meta)) { value <- meta$default } - if ((is.null(value) || value == '') && exists("type", where = meta)) { + description <- NA + if (exists("type", where = meta)) { + description <- paste("Value of", meta$type) + if (exists("pattern", where = meta)) { - value <- paste(PLACEHOLDER_TAG, "Value of", meta$type, "type with pattern:", meta$pattern) - } else { - value <- paste(PLACEHOLDER_TAG, "Value of", meta$type) + description <- paste(description, "type with pattern:", meta$pattern) + } + if (exists("format", where = meta)) { + description <- paste(description, "type with format:", meta$format) } } + if (is.null(value) || nchar(value) == 0) { + value <- paste(PLACEHOLDER_TAG, description) + } else { + comment <- description + } + if (exists("oneOf", where = meta) && length(meta$oneOf) > 0) { - value <- meta$oneOf[[1]]$const + possible_values <- sapply(meta$oneOf, function(x) x$const) + value <- possible_values[[1]] + + if (length(possible_values) == 1) { + possible_values <- I(list(list(possible_values))) + } else { + possible_values <- I(list(possible_values)) + } + + comment <- "One of" } - if (is.null(value) || nchar(value) == 0) NULL else value + data.frame(value = value, comment = comment, possible_values = possible_values) } -extract_template_param_array_default_value <- function(meta) { +extract_param_meta_for_number <- function(meta) { + value <- NULL + comment <- NA + + if (exists("default", where = meta)) { + value <- meta$default + } + + description <- "" + + if (exists("minimum", where = meta)) { + description <- paste0(description, "Min: ", meta$minimum, " ") + } + if (exists("maximum", where = meta)) { + description <- paste0(description, "Max: ", meta$maximum, " ") + } + + if (is.null(value) || nchar(value) == 0) { + value <- paste(PLACEHOLDER_TAG, description) + } else { + comment <- description + } + + data.frame(value = value, comment = comment, possible_values = NA) +} + +extract_param_meta_for_array <- function(meta) { + value <- NULL + if (exists("items", where = meta)) { if (exists("oneOf", meta$items) && length(meta$items$oneOf) > 0) { value <- sapply(meta$items$oneOf, function(x) { x$const }) - return(I(list(value))) + if (length(value) == 1) { + value <- I(list(list(value))) + } else { + value <- I(list(value)) + } } } - NULL + data.frame(value = I(value), comment = NA, possible_values = NA) } -extract_template_param_default_value <- function(meta) { +extract_param_metadata <- function(meta) { switch(meta$type, - "string" = extract_template_param_string_default_value(meta), - "array" = extract_template_param_array_default_value(meta) + "string" = extract_param_meta_for_string(meta), + "number" = extract_param_meta_for_number(meta), + "array" = extract_param_meta_for_array(meta), ) } strip_off_template_placeholders <- function(template) { output <- {} - t <- jsonlite::fromJSON(template) + t <- jsonlite::fromJSON(template, simplifyVector = FALSE) for (param in names(t)) { + if (startsWith(param, "_comment_")) next + if (startsWith(param, "_values_")) next value <- t[[param]] if (!is_placeholder(value)) { output[[param]] <- value } } - jsonlite::toJSON(output, pretty = TRUE, auto_unbox=TRUE) + jsonlite::toJSON(output, pretty = TRUE, auto_unbox = TRUE, digits = 17) } is_single_string <- function(input) { diff --git a/doc/hdar.R b/doc/hdar.R deleted file mode 100644 index 96e3b69..0000000 --- a/doc/hdar.R +++ /dev/null @@ -1,71 +0,0 @@ -## ----eval = FALSE------------------------------------------------------------- -# # Install hdar from CRAN (if available) or GitHub -# # install.packages("hdar") -# # or -# # devtools::install_github("eea/hdar@develop") -# -# # Load the hdar package -# library(hdar) - -## ----eval=FALSE--------------------------------------------------------------- -# # Define your username and password -# username <- "your_username" -# password <- "your_password" -# -# # Create an instance of the Client class and save credentials to a config file -# # The save_credentials parameter is optional and defaults to FALSE -# client <- Client$new(username, password, save_credentials = TRUE) - -## ----eval=FALSE--------------------------------------------------------------- -# # Create an instance of the Client class without passing credentials -# client <- Client$new() - -## ----eval=FALSE--------------------------------------------------------------- -# # Example method to check authentication by getting the auth token -# client$get_token() -# print(client$token()) - -## ----eval=FALSE--------------------------------------------------------------- -# # Assuming 'client' is already created and authenticated -# all_datasets <- client$datasets() -# print(all_datasets) - -## ----eval=FALSE--------------------------------------------------------------- -# # Assuming 'client' is already created and authenticated -# pattern <- "climate" -# filtered_datasets <- client$datasets(pattern) -# print(filtered_datase - -## ----eval = FALSE------------------------------------------------------------- -# # Assuming 'client' is already created and authenticated -# query_template <- client$generate_query_template("EO:CRYO:DAT:HRSI:FSC") -# print(query_template) - -## ----eval = FALSE------------------------------------------------------------- -# # Modify the query template as needed -# query_template$cloudCover <- "10" -# -# # Perform the search using the modified query template -# matches <- client$search(query_template) -# print(sapply(matches$results, function(x) { -# list( -# "id" = x$id, -# "size" = x$properties$size, -# "location" = x$properties$location -# ) -# })) - -## ----eval = FALSE------------------------------------------------------------- -# # Assuming 'client' is already created and authenticated, 'query' is defined -# matches <- client$search(query, 5) -# print(matches$results) - -## ----eval = FALSE------------------------------------------------------------- -# # Assuming 'matches' is an instance of SearchResults obtained from the search -# matches$download("~/downloads") - -## ----eval = FALSE------------------------------------------------------------- -# # Assuming 'matches' is an instance of SearchResults obtained from the search -# selected_indexes <- c(1, 3, 5) # Download only the 1st, 3rd, and 5th results -# matches$download("~/downloads", selected_indexes) - diff --git a/doc/hdar.Rmd b/doc/hdar.Rmd deleted file mode 100644 index 33bdd3c..0000000 --- a/doc/hdar.Rmd +++ /dev/null @@ -1,212 +0,0 @@ ---- -title: "A Guide to hdar package" -output: - rmarkdown::html_vignette: - toc: true - toc_depth: 2 -vignette: > - %\VignetteIndexEntry{A Guide to hdar package} - %\VignetteEngine{knitr::rmarkdown} - %\VignetteEncoding{UTF-8} ---- - -### Important Notice - -
- Notice: We are currently performing maintenance and improvements on the Backend service. You may experience intermittent slow responses or minor issues. Rest assured, our team is working hard to enhance your experience. Thank you for your patience! -
- -# Introduction - -The `hdar` R package provides seamless access to the WEkEO Harmonised Data Access (HDA) API, enabling users to efficiently query, download, and process data from the HDA platform. - -# Accessing the HDA Service - -To utilize the HDA service and library, you must first register for a WEkEO account. The HDA service is available at no cost to all WEkEO users. Creating an account allows you full access to our services, ensuring you can leverage the full capabilities of HDA seamlessly. Registration is straightforward and can be completed through the following link: [Register for WEkEO](https://www.wekeo.eu/register). Once your account is set up, you will be able to access the HDA services immediately. - -# Setup - -To start using the `hdar` package, you first need to install and load it in your R environment. - -```{r, eval = FALSE} -# Install hdar from CRAN (if available) or GitHub -# install.packages("hdar") -# or -# devtools::install_github("eea/hdar@develop") - -# Load the hdar package -library(hdar) -``` - -# Authentication - -To interact with the HDA service, you need to authenticate by providing your username and password. The `Client` class allows you to pass these credentials directly and optionally save them to a configuration file for future use. If credentials are not specified as parameters, the client will read them from the `~/.hdarc` file. - -## Creating and Authenticating the Client - -You can create an instance of the `Client` class by passing your username and password directly. TThe initialization method has an optional parameter `save_credentials` that specifies whether the provided credentials should be saved in the `~/.hdarc` configuration file. By default, `save_credential`s is set to `FALSE`. - -### Example: Pass User/Password at Initialization - -Here is an example of how to authenticate by passing the user and password, and optionally saving these credentials: - -```{r, eval=FALSE} -# Define your username and password -username <- "your_username" -password <- "your_password" - -# Create an instance of the Client class and save credentials to a config file -# The save_credentials parameter is optional and defaults to FALSE -client <- Client$new(username, password, save_credentials = TRUE) -``` - -If the `save_credentials` parameter is set to `TRUE`, the credentials will be saved in the `~/.hdarc` file, making it easier to authenticate in future sessions without passing the credentials again. - -### Example: Using Saved Credentials - -Once the credentials are saved, you can initialize the Client class without passing the credentials. The client will read the credentials from the `~/.hdarc` file: - -```{r, eval=FALSE} -# Create an instance of the Client class without passing credentials -client <- Client$new() -``` - -## Checking Authentication - -Once the client is created, you can check if it has been authenticated properly by calling a method `token()` that verifies authentication. For example: - -```{r, eval=FALSE} -# Example method to check authentication by getting the auth token -client$get_token() -print(client$token()) -``` - -By using one of these methods, you can securely authenticate with the HDA service and start making requests. - -# Finding Datasets - -To interact with the HDA service, you will often need to find datasets available on WEkEO. The Client class provides a method called `datasets` that lists available datasets, optionally filtered by a text pattern. - -## Basic Usage - -The basic usage of the datasets method is straightforward. You can retrieve a list of all datasets available on WEkEO by calling the `datasets` method on an instance of the `Client` class. - -```{r, eval=FALSE} -# Assuming 'client' is already created and authenticated -all_datasets <- client$datasets() -print(all_datasets) -``` - -## Filtering Datasets - -You can also filter the datasets by providing a text pattern. This is useful when you are looking for datasets that match a specific keyword or phrase. - -```{r, eval=FALSE} -# Assuming 'client' is already created and authenticated -pattern <- "climate" -filtered_datasets <- client$datasets(pattern) -print(filtered_datase -``` - -## Understanding the Results - -The datasets method returns a list containing datasets and associated information. This information may include dataset names, descriptions, and other metadata. - -# Creating a Query Template - -To search for data in the HDA service, you need to create a query template. Manually creating a query template can be tedious as it involves reading documentation and learning about possible parameters. To simplify this process, the `generate_query_template` function is provided to automate the creation of query templates for a given dataset. - -## Basic Usage - -The generate_query_template function generates a template of a query for a specified dataset. This function fetches information about existing parameters, default values, etc., from the `/queryable` endpoint of the HDA service. - -#### Example: Generating a Query Template - -Here is an example of how to generate a query template for the dataset with the ID "EO:CRYO:DAT:HRSI:FSC": - -```{r, eval = FALSE} -# Assuming 'client' is already created and authenticated -query_template <- client$generate_query_template("EO:CRYO:DAT:HRSI:FSC") -print(query_template) -``` - -## Understanding the Query Template - -The generated query template includes fields with descriptive placeholder values where default values are not specified. These placeholders provide hints about the expected value format. - -#### Example Query Template - -```{json} -{ - "dataset_id": "EO:CRYO:DAT:HRSI:FSC", - "cloudCover": "__### Value of string type with pattern: ^[0-9]*$", - "productIdentifier": "__### Value of string type with pattern: ^.*$", - "q": "__### Value of string type with pattern: ^.*$", - "observed_start": "2021-01-01T00:00:00.000Z", - "observed_end": "__### Value of string", - "published_start": "__### Value of string", - "published_end": "__### Value of string" -} -``` - -## Using the Generated Query Template - -You can and should customize the generated query template to fit your specific needs. You can remove any unnecessary parameters and set values for the ones you need. If you decide not to edit the placeholder values and use the generated query template directly in the search function, the fields with placeholder values (the ones starting with \_\_###) will be stripped off before sending the query to the HDA service. - -#### Example: Using the Query Template in a Search - -Here is an example of how to use the query template in a search: - -```{r, eval = FALSE} -# Modify the query template as needed -query_template$cloudCover <- "10" - -# Perform the search using the modified query template -matches <- client$search(query_template) -print(sapply(matches$results, function(x) { - list( - "id" = x$id, - "size" = x$properties$size, - "location" = x$properties$location - ) -})) -``` - -# Searching and Downloading Data - -To search for data in the HDA service, you can use the `search` function provided by the Client class. This function allows you to search for datasets based on a query and optionally limit the number of results. The search results can then be downloaded using the download method of the `SearchResults` class. - -## Searching for Data - -The `search` function takes a query and an optional limit parameter, which specifies the maximum number of results you want to retrieve. The function only searches for data and does not download it. The output of this function is an instance of the `SearchResults` class. - -#### Example: Searching for Data - -Here is an example of how to search for data using a query and limit the results to 5: - -```{r, eval = FALSE} -# Assuming 'client' is already created and authenticated, 'query' is defined -matches <- client$search(query, 5) -print(matches$results) -``` - -## Downloading the Found Results - -The `SearchResults` class has a public field results and a method called download that is responsible for downloading the found data. The `download()` function takes an output directory and an optional parameter to specify which results to download. - -#### Example: Downloading All Results - -```{r, eval = FALSE} -# Assuming 'matches' is an instance of SearchResults obtained from the search -matches$download("~/downloads") -``` - -#### Example: Downloading Selected Results - -You can also specify which results to download by providing their indexes: - -```{r, eval = FALSE} -# Assuming 'matches' is an instance of SearchResults obtained from the search -selected_indexes <- c(1, 3, 5) # Download only the 1st, 3rd, and 5th results -matches$download("~/downloads", selected_indexes) -``` diff --git a/doc/hdar.html b/doc/hdar.html deleted file mode 100644 index 8f44767..0000000 --- a/doc/hdar.html +++ /dev/null @@ -1,633 +0,0 @@ - - - - - - - - - - - - - - -A Guide to hdar package - - - - - - - - - - - - - - - - - - - - - - - - - - -

A Guide to hdar package

- - - - -
-

Important Notice

-
-

Notice: We are currently performing maintenance and -improvements on the Backend service. You may experience intermittent -slow responses or minor issues. Rest assured, our team is working hard -to enhance your experience. Thank you for your patience!

-
-
-
-

Introduction

-

The hdar R package provides seamless access to the WEkEO -Harmonised Data Access (HDA) API, enabling users to efficiently query, -download, and process data from the HDA platform.

-
-
-

Accessing the HDA Service

-

To utilize the HDA service and library, you must first register for a -WEkEO account. The HDA service is available at no cost to all WEkEO -users. Creating an account allows you full access to our services, -ensuring you can leverage the full capabilities of HDA seamlessly. -Registration is straightforward and can be completed through the -following link: Register for -WEkEO. Once your account is set up, you will be able to access the -HDA services immediately.

-
-
-

Setup

-

To start using the hdar package, you first need to -install and load it in your R environment.

-
# Install hdar from CRAN (if available) or GitHub
-# install.packages("hdar")
-# or
-# devtools::install_github("eea/hdar@develop")
-
-# Load the hdar package
-library(hdar)
-
-
-

Authentication

-

To interact with the HDA service, you need to authenticate by -providing your username and password. The Client class -allows you to pass these credentials directly and optionally save them -to a configuration file for future use. If credentials are not specified -as parameters, the client will read them from the ~/.hdarc -file.

-
-

Creating and Authenticating the Client

-

You can create an instance of the Client class by -passing your username and password directly. TThe initialization method -has an optional parameter save_credentials that specifies -whether the provided credentials should be saved in the -~/.hdarc configuration file. By default, -save_credentials is set to FALSE.

-
-

Example: Pass User/Password at Initialization

-

Here is an example of how to authenticate by passing the user and -password, and optionally saving these credentials:

-
# Define your username and password
-username <- "your_username"
-password <- "your_password"
-
-# Create an instance of the Client class and save credentials to a config file
-# The save_credentials parameter is optional and defaults to FALSE
-client <- Client$new(username, password, save_credentials = TRUE)
-

If the save_credentials parameter is set to -TRUE, the credentials will be saved in the -~/.hdarc file, making it easier to authenticate in future -sessions without passing the credentials again.

-
-
-

Example: Using Saved Credentials

-

Once the credentials are saved, you can initialize the Client class -without passing the credentials. The client will read the credentials -from the ~/.hdarc file:

-
# Create an instance of the Client class without passing credentials
-client <- Client$new()
-
-
-
-

Checking Authentication

-

Once the client is created, you can check if it has been -authenticated properly by calling a method token() that -verifies authentication. For example:

-
# Example method to check authentication by getting the auth token
-client$get_token()
-print(client$token())
-

By using one of these methods, you can securely authenticate with the -HDA service and start making requests.

-
-
-
-

Finding Datasets

-

To interact with the HDA service, you will often need to find -datasets available on WEkEO. The Client class provides a method called -datasets that lists available datasets, optionally filtered -by a text pattern.

-
-

Basic Usage

-

The basic usage of the datasets method is straightforward. You can -retrieve a list of all datasets available on WEkEO by calling the -datasets method on an instance of the Client -class.

-
# Assuming 'client' is already created and authenticated
-all_datasets <- client$datasets()
-print(all_datasets)
-
-
-

Filtering Datasets

-

You can also filter the datasets by providing a text pattern. This is -useful when you are looking for datasets that match a specific keyword -or phrase.

-
# Assuming 'client' is already created and authenticated
-pattern <- "climate"
-filtered_datasets <- client$datasets(pattern)
-print(filtered_datase
-
-
-

Understanding the Results

-

The datasets method returns a list containing datasets and associated -information. This information may include dataset names, descriptions, -and other metadata.

-
-
-
-

Creating a Query Template

-

To search for data in the HDA service, you need to create a query -template. Manually creating a query template can be tedious as it -involves reading documentation and learning about possible parameters. -To simplify this process, the generate_query_template -function is provided to automate the creation of query templates for a -given dataset.

-
-

Basic Usage

-

The generate_query_template function generates a template of a query -for a specified dataset. This function fetches information about -existing parameters, default values, etc., from the -/queryable endpoint of the HDA service.

-
-

Example: Generating a Query Template

-

Here is an example of how to generate a query template for the -dataset with the ID “EO:CRYO:DAT:HRSI:FSC”:

-
# Assuming 'client' is already created and authenticated
-query_template <- client$generate_query_template("EO:CRYO:DAT:HRSI:FSC")
-print(query_template)
-
-
-
-

Understanding the Query Template

-

The generated query template includes fields with descriptive -placeholder values where default values are not specified. These -placeholders provide hints about the expected value format.

-
-

Example Query Template

-
{
-  "dataset_id": "EO:CRYO:DAT:HRSI:FSC",
-  "cloudCover": "__### Value of string type with pattern: ^[0-9]*$",
-  "productIdentifier": "__### Value of string type with pattern: ^.*$",
-  "q": "__### Value of string type with pattern: ^.*$",
-  "observed_start": "2021-01-01T00:00:00.000Z",
-  "observed_end": "__### Value of string",
-  "published_start": "__### Value of string",
-  "published_end": "__### Value of string"
-}
-
-
-
-

Using the Generated Query Template

-

You can and should customize the generated query template to fit your -specific needs. You can remove any unnecessary parameters and set values -for the ones you need. If you decide not to edit the placeholder values -and use the generated query template directly in the search function, -the fields with placeholder values (the ones starting with __###) will -be stripped off before sending the query to the HDA service.

- -
-
-
-

Searching and Downloading Data

-

To search for data in the HDA service, you can use the -search function provided by the Client class. This function -allows you to search for datasets based on a query and optionally limit -the number of results. The search results can then be downloaded using -the download method of the SearchResults class.

-
-

Searching for Data

-

The search function takes a query and an optional limit -parameter, which specifies the maximum number of results you want to -retrieve. The function only searches for data and does not download it. -The output of this function is an instance of the -SearchResults class.

-
-

Example: Searching for Data

-

Here is an example of how to search for data using a query and limit -the results to 5:

-
# Assuming 'client' is already created and authenticated, 'query' is defined
-matches <- client$search(query, 5)
-print(matches$results)
-
-
-
-

Downloading the Found Results

-

The SearchResults class has a public field results and a -method called download that is responsible for downloading the found -data. The download() function takes an output directory and -an optional parameter to specify which results to download.

-
-

Example: Downloading All Results

-
# Assuming 'matches' is an instance of SearchResults obtained from the search
-matches$download("~/downloads")
-
-
-

Example: Downloading Selected Results

-

You can also specify which results to download by providing their -indexes:

-
# Assuming 'matches' is an instance of SearchResults obtained from the search
-selected_indexes <- c(1, 3, 5)  # Download only the 1st, 3rd, and 5th results
-matches$download("~/downloads", selected_indexes)
-
-
-
- - - - - - - - - - - diff --git a/hdar.Rproj b/hdar.Rproj index 0e13a5c..74e8894 100644 --- a/hdar.Rproj +++ b/hdar.Rproj @@ -18,4 +18,5 @@ StripTrailingWhitespace: Yes BuildType: Package PackageUseDevtools: Yes PackageInstallArgs: --no-multiarch --with-keep.source +PackageCheckArgs: --as-cran PackageRoxygenize: rd,collate,namespace diff --git a/man/Client.Rd b/man/Client.Rd index 7bb4b68..cb23e0a 100644 --- a/man/Client.Rd +++ b/man/Client.Rd @@ -177,13 +177,13 @@ List containing datasets and associated information. \subsection{Method \code{search()}}{ This function performs a search based on a specified query and returns an instance of \code{\link{SearchResults}}. \subsection{Usage}{ -\if{html}{\out{
}}\preformatted{Client$search(query, limit = NULL)}\if{html}{\out{
}} +\if{html}{\out{
}}\preformatted{Client$search(json_query, limit = NULL)}\if{html}{\out{
}} } \subsection{Arguments}{ \if{html}{\out{
}} \describe{ -\item{\code{query}}{Character string representing the search query.} +\item{\code{json_query}}{Character string representing the search query.} \item{\code{limit}}{Optional; a number specifying the maximum number of results to return.} } diff --git a/man/SearchResults.Rd b/man/SearchResults.Rd index 25aaaee..33e49a1 100644 --- a/man/SearchResults.Rd +++ b/man/SearchResults.Rd @@ -57,7 +57,12 @@ SearchResult instance \subsection{Method \code{download()}}{ Downloads resources based on stored results or selected indices of results. \subsection{Usage}{ -\if{html}{\out{
}}\preformatted{SearchResults$download(output_dir, selected_indexes, stop_at_failure = TRUE)}\if{html}{\out{
}} +\if{html}{\out{
}}\preformatted{SearchResults$download( + output_dir, + selected_indexes, + stop_at_failure = TRUE, + force = FALSE +)}\if{html}{\out{
}} } \subsection{Arguments}{ @@ -68,6 +73,8 @@ Downloads resources based on stored results or selected indices of results. \item{\code{selected_indexes}}{Optional; indices of the specific results to download.} \item{\code{stop_at_failure}}{Optional; controls whether the download process of multiple files should immediately stop upon encountering the first failure.} + +\item{\code{force}}{Optional; forces the download even if the file already exists in the specified output directory.} } \if{html}{\out{
}} } diff --git a/tests/testthat/test_auth.R b/tests/testthat/test_auth.R index fc076b5..1a948eb 100644 --- a/tests/testthat/test_auth.R +++ b/tests/testthat/test_auth.R @@ -1,20 +1,20 @@ library("testthat") library("hdar") -TOKEN_REGEXP = "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$" +TOKEN_REGEXP <- "^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$" test_that("Auth: incorrect credentials", { client <- Client$new("wrong_user", "wrong_pwd") expect_error(client$get_token()) - expect_equal(client$token(), NULL) + expect_error(client$token()) }) -#test_that("Auth: correct credentials", { +# test_that("Auth: correct credentials", { # client <- Client$new(USER, PWD) # expect_no_error(client$get_token()) # expect_match(client$token(), TOKEN_REGEXP) -#}) +# }) test_that("Auth: read credentials from file", { client <- Client$new() diff --git a/tests/testthat/test_query_template.R b/tests/testthat/test_query_template.R index 6bed867..e4b1b71 100644 --- a/tests/testthat/test_query_template.R +++ b/tests/testthat/test_query_template.R @@ -19,14 +19,11 @@ test_that("Template - Generate Query Template", { client <- Client$new() query_template <- client$generate_query_template("EO:CRYO:DAT:HRSI:FSC") - #print(query_template) + # print(query_template) - usable_query_template = strip_off_template_placeholders(query_template) - #print(usable_query_template) + usable_query_template <- strip_off_template_placeholders(query_template) + # print(usable_query_template) expect_true(check_if_placeholders_exist(query_template)) expect_false(check_if_placeholders_exist(usable_query_template)) }) - - - diff --git a/tests/testthat/test_search.R b/tests/testthat/test_search.R index 0dc84f4..afba833 100644 --- a/tests/testthat/test_search.R +++ b/tests/testthat/test_search.R @@ -1,12 +1,12 @@ library("testthat") library("hdar") -QUERY_CORRECT <- jsonlite::fromJSON('{ - "dataset_id": "EO:CRYO:DAT:HRSI:FSC", +QUERY_CORRECT <- '{ + "dataset_id": "EO:CRYO:DAT:HRSI:SWS", "observed_start": "2021-01-01T00:00:00.000Z" -}') +}' -QUERY_CORRECT2 <- jsonlite::fromJSON('{ +QUERY_CORRECT2 <- '{ "dataset_id": "EO:ECMWF:DAT:CAMS_GLOBAL_EMISSION_INVENTORIES", "variable": [ "acetaldehyde" @@ -21,9 +21,9 @@ QUERY_CORRECT2 <- jsonlite::fromJSON('{ "2000" ], "format": "zip" -}') +}' -QUERY_FAILED <- jsonlite::fromJSON('{ +QUERY_FAILED <- '{ "dataset_id": "EO:EEA:DAT:CLMS_HRVPP_VPP", "boundingBoxValues": [ { @@ -43,22 +43,30 @@ QUERY_FAILED <- jsonlite::fromJSON('{ "end": "2021-01-15T00:00:00.000Z" } ] -}') +}' test_that("Search - Matches Found", { client <- Client$new() - matches <- client$search(QUERY_CORRECT, 10) + matches <- client$search(QUERY_CORRECT, 2) + #print(matches) # Download files for *all* results temp_dir <- tempdir() matches$download(temp_dir) - print(list.files(temp_dir, recursive = TRUE)) + #print(list.files(temp_dir, recursive = TRUE)) + + # Test 'force' flag + expect_warning(matches$download(temp_dir), "File already exists:*") + expect_warning(matches$download(temp_dir, force = TRUE), NA) + # delete the folder and files unlink(temp_dir, recursive = TRUE, force = TRUE) + + expect_true(TRUE) }) -#test_that("Search - Failed Query", { -# client <- Client$new() -# expect_error(matches <- client$search(QUERY_FAILED), "Search query failed") -#}) +test_that("Search - Failed Query", { + client <- Client$new() + expect_error(matches <- client$search(QUERY_FAILED)) +}) diff --git a/tests/testthat/test_search_dataset.R b/tests/testthat/test_search_dataset.R index c33dd7e..972c08a 100644 --- a/tests/testthat/test_search_dataset.R +++ b/tests/testthat/test_search_dataset.R @@ -4,6 +4,7 @@ library("hdar") test_that("Search - Results", { client <- Client$new() found_datasets <- client$datasets() + # print(found_datasets) expect_no_error(found_datasets) expect_gte(length(found_datasets), 0) diff --git a/v1.0.0 b/v1.0.0 deleted file mode 100644 index e69de29..0000000 diff --git a/vignettes/WEkEO_UI_JSON_1.png b/vignettes/WEkEO_UI_JSON_1.png new file mode 100644 index 0000000..5c6573c Binary files /dev/null and b/vignettes/WEkEO_UI_JSON_1.png differ diff --git a/vignettes/WEkEO_UI_JSON_2.png b/vignettes/WEkEO_UI_JSON_2.png new file mode 100644 index 0000000..b77792a Binary files /dev/null and b/vignettes/WEkEO_UI_JSON_2.png differ diff --git a/vignettes/hdar.Rmd b/vignettes/hdar.Rmd index 7c5b784..56c6753 100644 --- a/vignettes/hdar.Rmd +++ b/vignettes/hdar.Rmd @@ -1,5 +1,5 @@ --- -title: "A Guide to hdar package" +title: "A Guide to the hdar package" output: rmarkdown::html_vignette: toc: true @@ -10,28 +10,22 @@ vignette: > %\VignetteEncoding{UTF-8} --- -### Important Notice - -
- Notice: We are currently performing maintenance and improvements on the Backend service. You may experience intermittent slow responses or minor issues. Rest assured, our team is working hard to enhance your experience. Thank you for your patience! -
- # Introduction -The `hdar` R package provides seamless access to the WEkEO Harmonised Data Access (HDA) API, enabling users to efficiently query, download, and process data from the HDA platform. +The `hdar` R package provides seamless access to the WEkEO Harmonised Data Access (HDA) API, enabling users to programmatically query and download data from within R. # Accessing the HDA Service -To utilize the HDA service and library, you must first register for a WEkEO account. The HDA service is available at no cost to all WEkEO users. Creating an account allows you full access to our services, ensuring you can leverage the full capabilities of HDA seamlessly. Registration is straightforward and can be completed through the following link: [Register for WEkEO](https://www.wekeo.eu/register). Once your account is set up, you will be able to access the HDA services immediately. +To utilize the HDA service and library, you must first register for a WEkEO account. Copernicus data and the HDA service are available at no cost to all WEkEO users. Creating an account allows you full access to our services, ensuring you can leverage the full capabilities of HDA seamlessly. Registration is straightforward and can be completed through the following link: [Register for WEkEO](https://www.wekeo.eu/register). Once your account is set up, you will be able to access the HDA services immediately. -# Setup +# Installation To start using the `hdar` package, you first need to install and load it in your R environment. ```{r, eval = FALSE} -# Install hdar from CRAN (if available) or GitHub -# install.packages("hdar") -# or +# Install stable hdar from CRAN +if(!require("hdar")){install.packages("hdar")} +# or develop version from GitHub # devtools::install_github("eea/hdar@develop") # Load the hdar package @@ -80,22 +74,62 @@ Once the client is created, you can check if it has been authenticated properly client$get_token() ``` -By using one of these methods, you can securely authenticate with the HDA service and start making requests. +# Copernicus Terms and Conditions (T&C) + +Copernicus data is free to use and modify, still T&Cs must be accepted in order to download the data. `hdarc` offers a confortable functionality to read and accept/reject T&C of the individual Copernicus service: + +```{r,eval=FALSE} +client$show_terms() +``` + +Will open a browser where you can read all the available T&Cs. +To accept/reject individual T&Cs or all at once use: + +```{r,eval=FALSE} +client$terms_and_conditions() + + term_id accepted +1 Copernicus_General_License FALSE +2 Copernicus_Sentinel_License FALSE +3 EUMETSAT_Core_Products_Licence FALSE +4 EUMETSAT_Copernicus_Data_Licence FALSE +5 Copernicus_DEM_Instance_COP-DEM-GLO-90-F_Global_90m FALSE +6 Copernicus_DEM_Instance_COP-DEM-GLO-30-F_Global_30m FALSE +7 Copernicus_ECMWF_License FALSE +8 Copernicus_Land_Monitoring_Service_Data_Policy FALSE +9 Copernicus_Marine_Service_Product_License FALSE +10 CNES_Open_2.0_ETALAB_Licence FALSE + +client$terms_and_conditions(term_id = 'all') + term_id accepted +1 Copernicus_General_License TRUE +2 Copernicus_Sentinel_License TRUE +3 EUMETSAT_Core_Products_Licence TRUE +4 EUMETSAT_Copernicus_Data_Licence TRUE +5 Copernicus_DEM_Instance_COP-DEM-GLO-90-F_Global_90m TRUE +6 Copernicus_DEM_Instance_COP-DEM-GLO-30-F_Global_30m TRUE +7 Copernicus_ECMWF_License TRUE +8 Copernicus_Land_Monitoring_Service_Data_Policy TRUE +9 Copernicus_Marine_Service_Product_License TRUE +10 CNES_Open_2.0_ETALAB_Licence TRUE + +``` # Finding Datasets -To interact with the HDA service, you will often need to find datasets available on WEkEO. The Client class provides a method called `datasets` that lists available datasets, optionally filtered by a text pattern. +WEkEO offers a vast amount of different products. To find what you need the Client class provides a method called `datasets` that lists available datasets, optionally filtered by a text pattern. ## Basic Usage The basic usage of the datasets method is straightforward. You can retrieve a list of all datasets available on WEkEO by calling the `datasets` method on an instance of the `Client` class. ```{r, eval=FALSE} -# Assuming 'client' is already created and authenticated -# client <- Client$new() +# retrieving the full list +# This takes about 2 minutes! all_datasets <- client$datasets() -all_datasets -all_datasets + +# list all datasets IDs on WEkEO +sapply(filtered_datasets,FUN = function(x){x$dataset_id}) ``` ## Filtering Datasets @@ -103,116 +137,149 @@ all_datasets You can also filter the datasets by providing a text pattern. This is useful when you are looking for datasets that match a specific keyword or phrase. ```{r, eval=FALSE} -# Assuming 'client' is already created and authenticated -# client <- Client$new() - -pattern <- "Seasonal Trajectories" -# client <- Client$new() -pattern <- "Seasonal Trajectories" -filtered_datasets <- client$datasets(pattern) +filtered_datasets <- client$datasets("Seasonal Trajectories") # list dataset IDs sapply(filtered_datasets,FUN = function(x){x$dataset_id}) [1] "EO:EEA:DAT:CLMS_HRVPP_VPP-LAEA" "EO:EEA:DAT:CLMS_HRVPP_ST" "EO:EEA:DAT:CLMS_HRVPP_ST-LAEA" [4] "EO:EEA:DAT:CLMS_HRVPP_VPP" + + +filtered_datasets <- client$datasets("Baltic") + +# list dataset IDs +sapply(filtered_datasets,FUN = function(x){x$dataset_id}) + [1] "EO:MO:DAT:BALTICSEA_ANALYSISFORECAST_BGC_003_007:cmems_mod_bal_bgc-pp_anfc_P1D-i_202311" + [2] "EO:MO:DAT:NWSHELF_MULTIYEAR_PHY_004_009:cmems_mod_nws_phy-sst_my_7km-2D_PT1H-i_202112" + [3] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L4_MY_009_134:cmems_obs-oc_bal_bgc-plankton_my_l4-multi-1km_P1M_202211" + [4] "EO:MO:DAT:SST_BAL_PHY_SUBSKIN_L4_NRT_010_034:cmems_obs-sst_bal_phy-subskin_nrt_l4_PT1H-m_202211" + [5] "EO:MO:DAT:BALTICSEA_MULTIYEAR_PHY_003_011:cmems_mod_bal_phy_my_P1Y-m_202303" + [6] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L3_NRT_009_131:cmems_obs-oc_bal_bgc-transp_nrt_l3-olci-300m_P1D_202207" + [7] "EO:MO:DAT:BALTICSEA_MULTIYEAR_BGC_003_012:cmems_mod_bal_bgc_my_P1Y-m_202303" + [8] "EO:MO:DAT:SST_BAL_SST_L4_REP_OBSERVATIONS_010_016:DMI_BAL_SST_L4_REP_OBSERVATIONS_010_016_202012" + [9] "EO:MO:DAT:BALTICSEA_ANALYSISFORECAST_PHY_003_006:cmems_mod_bal_phy_anfc_PT15M-i_202311" +[10] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L3_MY_009_133:cmems_obs-oc_bal_bgc-plankton_my_l3-multi-1km_P1D_202207" +[11] "EO:MO:DAT:SST_BAL_PHY_L3S_MY_010_040:cmems_obs-sst_bal_phy_my_l3s_P1D-m_202211" +[12] "EO:MO:DAT:SEAICE_BAL_SEAICE_L4_NRT_OBSERVATIONS_011_004:FMI-BAL-SEAICE_THICK-L4-NRT-OBS" +[13] "EO:MO:DAT:SEAICE_BAL_PHY_L4_MY_011_019:cmems_obs-si_bal_seaice-conc_my_1km_202112" +[14] "EO:MO:DAT:BALTICSEA_ANALYSISFORECAST_WAV_003_010:cmems_mod_bal_wav_anfc_PT1H-i_202311" +[15] "EO:MO:DAT:BALTICSEA_REANALYSIS_WAV_003_015:dataset-bal-reanalysis-wav-hourly_202003" +[16] "EO:MO:DAT:OCEANCOLOUR_BAL_BGC_L4_NRT_009_132:cmems_obs-oc_bal_bgc-plankton_nrt_l4-olci-300m_P1M_202207" +[17] "EO:MO:DAT:SST_BAL_SST_L3S_NRT_OBSERVATIONS_010_032:DMI-BALTIC-SST-L3S-NRT-OBS_FULL_TIME_SERIE_201904" + ``` ## Understanding the Results The datasets method returns a list containing datasets and associated information. This information may include dataset names, descriptions, and other metadata. -# Creating a Query Template - -To search for data in the HDA service, you need to create a query template. Manually creating a query template can be tedious as it involves reading documentation and learning about possible parameters. To simplify this process, the `generate_query_template` function is provided to automate the creation of query templates for a given dataset. +```{r, eval=FALSE} +client$datasets("EO:ECMWF:DAT:DERIVED_NEAR_SURFACE_METEOROLOGICAL_VARIABLES") +[[1]] +[[1]]$terms +[[1]]$terms[[1]] +[1] "Copernicus_ECMWF_License" -## Basic Usage +[[1]]$dataset_id +[1] "EO:ECMWF:DAT:DERIVED_NEAR_SURFACE_METEOROLOGICAL_VARIABLES" -The `generate_query_template` function generates a template of a query for a specified dataset. This function fetches information about existing parameters, default values, etc., from the `/queryable` endpoint of the HDA service. -The `generate_query_template` function generates a template of a query for a specified dataset. This function fetches information about existing parameters, default values, etc., from the `/queryable` endpoint of the HDA service. +[[1]]$title +[1] "Near surface meteorological variables from 1979 to 2019 derived from bias-corrected reanalysis" -#### Example: Generating a Query +[[1]]$abstract +[1] "This dataset provides bias-corrected reconstruction of near-surface meteorological variables derived from the fifth generation of the European Centre for Medium-Range Weather Forecasts (ECMWF) atmospheric reanalyses (ERA5). It is intended to be used as a meteorological forcing dataset for land surface and hydrological models. \nThe dataset has been obtained using the same methodology used to derive the widely used water, energy and climate change (WATCH) forcing data, and is thus also referred to as WATCH Forcing Data methodology applied to ERA5 (WFDE5). The data are derived from the ERA5 reanalysis product that have been re-gridded to a half-degree resolution. Data have been adjusted using an elevation correction and monthly-scale bias corrections based on Climatic Research Unit (CRU) data (for temperature, diurnal temperature range, cloud-cover, wet days number and precipitation fields) and Global Precipitation Climatology Centre (GPCC) data (for precipitation fields only). Additional corrections are included for varying atmospheric aerosol-loading and separate precipitation gauge observations. For full details please refer to the product user-guide.\nThis dataset was produced on behalf of Copernicus Climate Change Service (C3S) and was generated entirely within the Climate Data Store (CDS) Toolbox. The toolbox source code is provided in the documentation tab.\n\nVariables in the dataset/application are:\nGrid-point altitude, Near-surface air temperature, Near-surface specific humidity, Near-surface wind speed, Rainfall flux, Snowfall flux, Surface air pressure, Surface downwelling longwave radiation, Surface downwelling shortwave radiation" -#### Example: Generating a Query +[[1]]$doi +NULL -Here is an example of how to generate a query template for the dataset with the ID "EO:EEA:DAT:CLMS_HRVPP_ST": -Here is an example of how to generate a query template for the dataset with the ID "EO:EEA:DAT:CLMS_HRVPP_ST": +[[1]]$thumbnails +[1] "https://datastore.copernicus-climate.eu/c3s/published-forms-v2/c3sprod/derived-near-surface-meteorological-variables/overview.jpg" -```{r, eval = FALSE} -# client <- Client$new() -query_template <- client$generate_query_template("EO:EEA:DAT:CLMS_HRVPP_ST") -query_template -{ - "dataset_id": "EO:EEA:DAT:CLMS_HRVPP_ST", - "uid": "__### Value of string type with pattern: [\\w-]+", - "productType": "PPI", - "platformSerialIdentifier": "S2A, S2B", - "tileId": "__### Value of string type with pattern: [\\w-]+", - "productVersion": "__### Value of string type with pattern: [\\w-]+", - "resolution": "10", - "processingDate": "__### Value of string", - "start": "__### Value of string", - "end": "__### Value of string", - "bbox": [ - -180, - -90, - 180, - 90 - ] -} ``` -```{r, eval = FALSE} -# convert to list for easier manipulation in R -library(jsonlite) -query_template <- fromJSON(query_template) -query_template -$dataset_id -[1] "EO:EEA:DAT:CLMS_HRVPP_ST" +# Creating a Query Template -$uid -[1] "__### Value of string type with pattern: [\\w-]+" +To search for a specific product, you need to create a query template. You can either use the [WEkEO viewer](https://www.wekeo.eu/data?view=viewer) and copy paste the JSON query: -$productType -[1] "PPI" +```{r, fig.show='hold', echo=FALSE,out.width="45%"} +knitr::include_graphics(c("WEkEO_UI_JSON_1.png","WEkEO_UI_JSON_2.png")) -$platformSerialIdentifier -[1] "S2A, S2B" +``` -$tileId -[1] "__### Value of string type with pattern: [\\w-]+" +```{r,echo=TRUE} +query <- '{ + "dataset_id": "EO:ECMWF:DAT:CEMS_GLOFAS_HISTORICAL", + "system_version": [ + "version_4_0" + ], + "hydrological_model": [ + "lisflood" + ], + "product_type": [ + "consolidated" + ], + "variable": [ + "river_discharge_in_the_last_24_hours" + ], + "hyear": [ + "2024" + ], + "hmonth": [ + "june" + ], + "hday": [ + "01" + ], + "format": "grib", + "bbox": [ + 11.77115199576009, + 44.56907885098417, + 13.0263737724595, + 45.40384015467251 + ], + "itemsPerPage": 200, + "startIndex": 0 +}' +``` -$productVersion -[1] "__### Value of string type with pattern: [\\w-]+" +or use the query template function `generate_query_template` for a given dataset. -$resolution -[1] "10" +## Basic Usage -$processingDate -[1] "__### Value of string" +The `generate_query_template` function generates a template of a query for a specified dataset. This function fetches information about existing parameters, default values, etc., from the `/queryable` endpoint of the HDA service. -$start -[1] "__### Value of string" +#### Example: Generating a Query -$end -[1] "__### Value of string" +Here is an example of how to generate a query template for the dataset with the ID "EO:EEA:DAT:CLMS_HRVPP_ST": -$bbox -[1] -180 -90 180 90 +```{r, eval = FALSE} # client <- Client$new() query_template <- client$generate_query_template("EO:EEA:DAT:CLMS_HRVPP_ST") query_template { "dataset_id": "EO:EEA:DAT:CLMS_HRVPP_ST", + "itemsPerPage": 11, + "startIndex": 0, "uid": "__### Value of string type with pattern: [\\w-]+", "productType": "PPI", + "_comment_productType": "One of", + "_values_productType": ["PPI", "QFLAG"], "platformSerialIdentifier": "S2A, S2B", + "_comment_platformSerialIdentifier": "One of", + "_values_platformSerialIdentifier": [ + "S2A, S2B" + ], "tileId": "__### Value of string type with pattern: [\\w-]+", "productVersion": "__### Value of string type with pattern: [\\w-]+", "resolution": "10", - "processingDate": "__### Value of string", - "start": "__### Value of string", - "end": "__### Value of string", + "_comment_resolution": "One of", + "_values_resolution": [ + "10" + ], + "processingDate": "__### Value of string type with format: date-time", + "start": "__### Value of string type with format: date-time", + "end": "__### Value of string type with format: date-time", "bbox": [ -180, -90, @@ -222,23 +289,51 @@ query_template } ``` +## Modify and use the generated Query Template + +You can and should customize the generated query template to fit your specific needs. Fields starting with `__###` are placeholders indicating possible values. If these placeholders are left unchanged, they will be automatically removed before sending the query to the HDA service. Additionally, fields with the prefix `_comment_` provide relevant information regarding the specified field, such as possible values, format, or data patterns. Like the placeholders, these comment fields will also be automatically removed before the query is sent. + +Placeholders are used when there is no way to derive the value from the metadata endpoint, while comment fields appear when the field has a value already defined, offering additional context for customizing the query. + +Furthermore, fields prefixed with `_values_` contain all possible values for a specific field. This allows you to programmatically reference them in your code with ease, simplifying customization and ensuring that you have access to valid options when configuring the query. + +To modify the query, it is often easier to transform the JSON into an R list using the `jsonlite::fromJSON()` function: + ```{r, eval = FALSE} # convert to list for easier manipulation in R library(jsonlite) -query_template <- fromJSON(query_template) +query_template <- fromJSON(query_template, flatten = FALSE) query_template $dataset_id [1] "EO:EEA:DAT:CLMS_HRVPP_ST" +$itemsPerPage +[1] 11 + +$startIndex +[1] 0 + $uid [1] "__### Value of string type with pattern: [\\w-]+" $productType [1] "PPI" +$`_comment_productType` +[1] "One of" + +$`_values_productType` +[1] "PPI" "QFLAG" + $platformSerialIdentifier [1] "S2A, S2B" +$`_comment_platformSerialIdentifier` +[1] "One of" + +$`_values_platformSerialIdentifier` +[1] "S2A, S2B" + $tileId [1] "__### Value of string type with pattern: [\\w-]+" @@ -248,26 +343,25 @@ $productVersion $resolution [1] "10" +$`_comment_resolution` +[1] "One of" + +$`_values_resolution` +[1] "10" + $processingDate -[1] "__### Value of string" +[1] "__### Value of string type with format: date-time" $start -[1] "__### Value of string" +[1] "__### Value of string type with format: date-time" $end -[1] "__### Value of string" +[1] "__### Value of string type with format: date-time" $bbox [1] -180 -90 180 90 ``` -## Modify and use the generated Query Template - -## Modify and use the generated Query Template - -You can and should customize the generated query template to fit your specific needs. Fields starting with '\_\_###' are placeholder values indicating possible values, but if unchanged it will be stripped off before sending the query to the HDA service. -You can and should customize the generated query template to fit your specific needs. Fields starting with '\_\_###' are placeholder values indicating possible values, but if unchanged it will be stripped off before sending the query to the HDA service. - Here is an example of how to use the query template in a search: ```{r, eval = FALSE} @@ -278,55 +372,35 @@ query_template$bbox <- c(11.1090, 46.6210, 11.2090, 46.7210) query_template$start <- "2018-03-01T00:00:00.000Z" query_template$end <- "2018-05-31T00:00:00.000Z" query_template + $dataset_id [1] "EO:EEA:DAT:CLMS_HRVPP_ST" +$itemsPerPage +[1] 11 + +$startIndex +[1] 0 + $uid [1] "__### Value of string type with pattern: [\\w-]+" $productType [1] "PPI" -$platformSerialIdentifier -[1] "S2A, S2B" +$`_comment_productType` +[1] "One of" -$tileId -[1] "__### Value of string type with pattern: [\\w-]+" +$`_values_productType` +[1] "PPI" "QFLAG" -$productVersion -[1] "__### Value of string type with pattern: [\\w-]+" - -$resolution -[1] "10" - -$processingDate -[1] "__### Value of string" - -$start -[1] "2018-03-01T00:00:00.000Z" - -$end -[1] "2018-05-31T00:00:00.000Z" - -$bbox -[1] 11.109 46.621 11.209 46.721 -# set a new bbox -query_template$bbox <- c(11.1090, 46.6210, 11.2090, 46.7210) - -# limit the time range -query_template$start <- "2018-03-01T00:00:00.000Z" -query_template$end <- "2018-05-31T00:00:00.000Z" -query_template -$dataset_id -[1] "EO:EEA:DAT:CLMS_HRVPP_ST" - -$uid -[1] "__### Value of string type with pattern: [\\w-]+" +$platformSerialIdentifier +[1] "S2A, S2B" -$productType -[1] "PPI" +$`_comment_platformSerialIdentifier` +[1] "One of" -$platformSerialIdentifier +$`_values_platformSerialIdentifier` [1] "S2A, S2B" $tileId @@ -338,8 +412,14 @@ $productVersion $resolution [1] "10" +$`_comment_resolution` +[1] "One of" + +$`_values_resolution` +[1] "10" + $processingDate -[1] "__### Value of string" +[1] "__### Value of string type with format: date-time" $start [1] "2018-03-01T00:00:00.000Z" @@ -351,6 +431,13 @@ $bbox [1] 11.109 46.621 11.209 46.721 ``` +Once you have made the necessary modifications, you can convert the list back to JSON format with the `jsonlite::toJSON()` function. It's crucial to use the `auto_unbox = TRUE` flag when converting back to JSON. This ensures that the JSON is correctly formatted, particularly for arrays with a single element, due to the way `jsonlite` handles serialization. + +```{r, eval = FALSE} +# convert to JSON format +query_template <- toJSON(query_template, auto_unbox = TRUE, digits = 17) # don't forget to put auto_unbox = TRUE +``` + # Searching and Downloading Data To search for data in the HDA service, you can use the `search` function provided by the Client class. This function allows you to search for datasets based on a query and optionally limit the number of results. The search results can then be downloaded using the download method of the `SearchResults` class. @@ -367,16 +454,6 @@ matches <- client$search(query_template) [1] "Found 9 files" [1] "Total Size 1.8 GB" -sapply(matches$results,FUN = function(x){x$id}) -[1] "ST_20180301T000000_S2_T32TPS-010m_V101_PPI" "ST_20180311T000000_S2_T32TPS-010m_V101_PPI" -[3] "ST_20180321T000000_S2_T32TPS-010m_V101_PPI" "ST_20180401T000000_S2_T32TPS-010m_V101_PPI" -[5] "ST_20180411T000000_S2_T32TPS-010m_V101_PPI" "ST_20180421T000000_S2_T32TPS-010m_V101_PPI" -[7] "ST_20180501T000000_S2_T32TPS-010m_V101_PPI" "ST_20180511T000000_S2_T32TPS-010m_V101_PPI" -[9] "ST_20180521T000000_S2_T32TPS-010m_V101_PPI" -matches <- client$search(query_template) -[1] "Found 9 files" -[1] "Total Size 1.8 GB" - sapply(matches$results,FUN = function(x){x$id}) [1] "ST_20180301T000000_S2_T32TPS-010m_V101_PPI" "ST_20180311T000000_S2_T32TPS-010m_V101_PPI" [3] "ST_20180321T000000_S2_T32TPS-010m_V101_PPI" "ST_20180401T000000_S2_T32TPS-010m_V101_PPI" @@ -387,8 +464,7 @@ sapply(matches$results,FUN = function(x){x$id}) ## Downloading the files -The `SearchResults` class has a public field results and a method called download that is responsible for downloading the found data. The `download()` function takes an output directory (which is created if no existing) and the optional parameter `selected_indexes` to specify which files to download. -The `SearchResults` class has a public field results and a method called download that is responsible for downloading the found data. The `download()` function takes an output directory (which is created if no existing) and the optional parameter `selected_indexes` to specify which files to download. +The `SearchResults` class has a public field `results` and a method called `download` that is responsible for downloading the found data. The `download()` function takes an output directory (which is created if it doesn't already exist) and includes an optional `force` parameter. When `force` is set to `TRUE`, the function will re-download the files even if they already exist in the output directory, overwriting the existing files. If `force` is set to `FALSE` (the default), the function will skip downloading files that already exist, saving time and bandwidth. ```{r, eval = FALSE} # Assuming 'matches' is an instance of SearchResults obtained from the search @@ -416,13 +492,5 @@ list.files(odir) [7] "ST_20180501T000000_S2_T32TPS-010m_V101_PPI.tif" "ST_20180511T000000_S2_T32TPS-010m_V101_PPI.tif" [9] "ST_20180521T000000_S2_T32TPS-010m_V101_PPI.tif" -unlink(odir,recursive = TRUE) -list.files(odir) -[1] "ST_20180301T000000_S2_T32TPS-010m_V101_PPI.tif" "ST_20180311T000000_S2_T32TPS-010m_V101_PPI.tif" -[3] "ST_20180321T000000_S2_T32TPS-010m_V101_PPI.tif" "ST_20180401T000000_S2_T32TPS-010m_V101_PPI.tif" -[5] "ST_20180411T000000_S2_T32TPS-010m_V101_PPI.tif" "ST_20180421T000000_S2_T32TPS-010m_V101_PPI.tif" -[7] "ST_20180501T000000_S2_T32TPS-010m_V101_PPI.tif" "ST_20180511T000000_S2_T32TPS-010m_V101_PPI.tif" -[9] "ST_20180521T000000_S2_T32TPS-010m_V101_PPI.tif" - unlink(odir,recursive = TRUE) ```