Fix R-CMD-CHECK workflow ubuntu issues #39

annakrystalli · 2022-02-05T13:48:43Z

Looking into the failures in the ubuntu runs on the R-CMD-CHECK workflow. In the end, it's probably not an issue with sf itself but changes in the testing environment when moving to 20.04. The errors during sf install point to it potentially being a problem with configurations arising from multiple versions of proj.

Experimenting with whether the update command might be causing multiple copies.

The test-services workflow was not in fact failing on sf installation which was successful, and the only Linux configuration details I can see between the two setups I can see is that test-services did not include an update step.

Instead, test-services was failing on the installation of ows4r because of error in installation of sodium (see https://github.com/EMODnet/EMODnetWFS/runs/5045539315?check_suite_focus=true#step:6:2764)

* installing *source* package ‘sodium’ ...
** package ‘sodium’ successfully unpacked and MD5 sums checked
** using staged installation
Package libsodium was not found in the pkg-config search path.
Perhaps you should add the directory containing `libsodium.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libsodium' found
Using PKG_CFLAGS=
Using PKG_LIBS=-lsodium
--------------------------- [ANTICONF] --------------------------------
Configuration failed because libsodium was not found. Try installing:
 * deb: libsodium-dev (Debian, Ubuntu, etc)
 * rpm: libsodium-devel (Fedora, EPEL)
 * csw: libsodium_dev (Solaris)
 * brew: libsodium (OSX)
If libsodium is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a libsodium.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
-------------------------- [ERROR MESSAGE] ---------------------------
<stdin>:1:10: fatal error: sodium.h: No such file or directory
compilation terminated.
--------------------------------------------------------------------
ERROR: configuration failed for package ‘sodium’
* removing ‘/home/runner/work/_temp/Library/sodium’

which leads to a cascade of missing packages affecting ows4r https://github.com/EMODnet/EMODnetWFS/runs/5045539315?check_suite_focus=true#step:6:4853

Warning messages:
1: In i.p(...) : installation of package ‘sodium’ had non-zero exit status
2: In i.p(...) :
  installation of package ‘keyring’ had non-zero exit status
3: In i.p(...) :
  installation of package ‘geometa’ had non-zero exit status
4: In i.p(...) : installation of package ‘ows4R’ had non-zero exit status

Looks like the issue started through adding keyring as a geometa dependency in https://github.com/eblondel/geometa/blob/d22371e78a1367174c4437d7f9915e66cc332346/DESCRIPTION, 12 days ago, which is roughly when the errors started appearing in Test-Services.

In any case, I've added libsodium-dev as a system dependency for Linux.

Will now test to see whether the not updating solves gdal and proj configuration issues. 🤞

codecov-commenter · 2022-02-05T14:14:24Z

Codecov Report

Merging #39 (a504f4a) into master (fcab4f6) will not change coverage.
The diff coverage is n/a.

@@           Coverage Diff           @@
##           master      #39   +/-   ##
=======================================
  Coverage   84.86%   84.86%           
=======================================
  Files           4        4           
  Lines         218      218           
=======================================
  Hits          185      185           
  Misses         33       33

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update fcab4f6...a504f4a. Read the comment docs.

annakrystalli · 2022-02-05T14:23:29Z

So everything installs correctly now! 💪

However new errors introduced now in all ubuntu runs!! 😭 https://github.com/EMODnet/EMODnetWFS/runs/5078071611?check_suite_focus=true#step:13:31

Quitting from lines 176-177 (eqcl_filtering.Rmd) 
Error: Error: processing vignette 'eqcl_filtering.Rmd' failed with diagnostics:
error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
--- failed re-building ‘eqcl_filtering.Rmd’
SUMMARY: processing the following file failed:
  ‘eqcl_filtering.Rmd’
Error: Error: Vignette re-building failed.
Execution halted
Error: Error in proc$get_built_file() : Build process failed
Calls: <Anonymous> ... build_package -> with_envvar -> force -> <Anonymous>
Execution halted
Error: Process completed with exit code 1.

The offending lines in the vignette are:

EMODnetWFS/vignettes/eqcl_filtering.Rmd

Lines 175 to 177 in fcab4f6

    
           ```{r} 
        
           wfs_gs <- emodnet_init_wfs_client(service = "geology_seabed_substrate_maps") 
        
           ```

🤷‍♀️ have no idea why so more digging needed

annakrystalli · 2022-02-05T14:54:22Z

This may be an issue with setup of libcurl4-openssl-dev in testing environment or a problem with the signatures sent back by the server. If it is that, one way around it would be to put SECLEVEL 1 in /etc/ssl/openssl.cnf to allow for insecure outdated signatures. But again, that's insecure and outdated.

FreshRSS/FreshRSS#3029 (comment)

More context about server signatures and this error here: openssl/openssl#7126 (comment)

maelle · 2022-03-01T14:08:05Z

In #40 I am trying using the latest r-lib/actions workflows (that are much shorter so easier to read & tweak).

maelle · 2022-03-01T14:34:40Z

Which gets me to the same state at this PR. I'm now running things locally. The new error might be a server problem so to be reported. 🤔

maelle · 2022-03-01T14:42:37Z

yay I can reproduce the error locally :-)

maelle · 2022-03-01T14:44:43Z

curl -v -X GET https://drive.emodnet-geology.eu/geoserver/gtk/wfs
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 195.41.77.182:443...
* TCP_NODELAY set
* Connected to drive.emodnet-geology.eu (195.41.77.182) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (OUT), TLS alert, handshake failure (552):
* error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
* Closing connection 0
curl: (35) error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type

maelle · 2022-03-01T14:46:17Z

and with another URL (that works)

curl -v -X GET https://ows.emodnet-humanactivities.eu/wfs
Note: Unnecessary use of -X or --request, GET is already inferred.
*   Trying 217.160.174.61:443...
* TCP_NODELAY set
* Connected to ows.emodnet-humanactivities.eu (217.160.174.61) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):

maelle · 2022-03-01T14:56:14Z

urls <- EMODnetWFS::emodnet_wfs$service_url

check_url <- function(url) {
    try <- try(httr2::request(url) |>
    httr2::req_perform())
    if (inherits(try, "try-error")) print(url)
}

purrr::walk(urls, check_url)
#> Error in resp_check_status(resp, error_body(req, resp)) : 
#>   HTTP 400 Bad Request.
#> [1] "https://ows.emodnet-bathymetry.eu/wfs"
#> Error in resp_check_status(resp, error_body(req, resp)) : 
#>   HTTP 400 Bad Request.
#> [1] "https://nodc.ogs.trieste.it/geoserver/Contaminants/wfs"
#> Error : error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/tno/wfs"
#> Error : error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/ispra/wfs"
#> Error : error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/gsi/wfs"
#> Error : error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/bgr/wfs"
#> Error : error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/gtk/wfs"
#> Error : error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/bgs/wfs"
#> Error in resp_check_status(resp, error_body(req, resp)) : 
#>   HTTP 400 Bad Request.
#> [1] "https://ows.emodnet-humanactivities.eu/wfs"
#> Error in resp_check_status(resp, error_body(req, resp)) : 
#>   HTTP 400 Bad Request.
#> [1] "https://geoserver.emodnet-physics.eu/geoserver/emodnet/wfs"
#> Error in resp_check_status(resp, error_body(req, resp)) : 
#>   HTTP 400 Bad Request.
#> [1] "https://ows.emodnet-seabedhabitats.eu/geoserver/emodnet_open/wfs"
#> Error in resp_check_status(resp, error_body(req, resp)) : 
#>   HTTP 400 Bad Request.
#> [1] "https://ows.emodnet-seabedhabitats.eu/geoserver/emodnet_open_maplibrary/wfs"

^{Created on 2022-03-01 by the reprex package (v2.0.1)}

maelle · 2022-03-01T15:15:04Z

The actually problematic ones (no client inits possible) are the drive dot emodnet

maelle · 2022-03-01T15:21:28Z

urls <- EMODnetWFS::emodnet_wfs$service_url
urls <- urls[grepl("drive\\.emodnet-geology", urls)]
urls
#> [1] "https://drive.emodnet-geology.eu/geoserver/tno/wfs"  
#> [2] "https://drive.emodnet-geology.eu/geoserver/ispra/wfs"
#> [3] "https://drive.emodnet-geology.eu/geoserver/gsi/wfs"  
#> [4] "https://drive.emodnet-geology.eu/geoserver/bgr/wfs"  
#> [5] "https://drive.emodnet-geology.eu/geoserver/gtk/wfs"  
#> [6] "https://drive.emodnet-geology.eu/geoserver/bgs/wfs"

check_url <- function(url, service_version = "2.0.0") {
    try <- try(
        EMODnetWFS:::perform_http_request(url)
    )
    
    if (inherits(try, "try-error")) print(url)
}

purrr::walk(urls, check_url)
#> x WFS client creation failed.
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/tno/wfs'
#> Error in curl::curl_fetch_memory(url, handle = handle) : 
#>   error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/tno/wfs"
#> x WFS client creation failed.
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/ispra/wfs'
#> Error in curl::curl_fetch_memory(url, handle = handle) : 
#>   error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/ispra/wfs"
#> x WFS client creation failed.
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/gsi/wfs'
#> Error in curl::curl_fetch_memory(url, handle = handle) : 
#>   error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/gsi/wfs"
#> x WFS client creation failed.
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/bgr/wfs'
#> Error in curl::curl_fetch_memory(url, handle = handle) : 
#>   error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/bgr/wfs"
#> x WFS client creation failed.
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/gtk/wfs'
#> Error in curl::curl_fetch_memory(url, handle = handle) : 
#>   error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/gtk/wfs"
#> x WFS client creation failed.
#> ℹ Service: 'https://drive.emodnet-geology.eu/geoserver/bgs/wfs'
#> Error in curl::curl_fetch_memory(url, handle = handle) : 
#>   error:1414D172:SSL routines:tls12_check_peer_sigalg:wrong signature type
#> [1] "https://drive.emodnet-geology.eu/geoserver/bgs/wfs"

^{Created on 2022-03-01 by the reprex package (v2.0.1)}

maelle · 2022-03-01T16:11:06Z

https://wizardzines.com/comics/certificates/

maelle · 2022-03-11T08:13:23Z

See eblondel/ows4R#64

maelle · 2022-03-11T08:24:33Z

Until we can use the curl hack via ows4r or the web services are fixed, how about we drop the web services from the package for Linux and open an issue?

The other possibility would be to write some sort of workaround for not having to wait for a decision in ows4r but that does not seem like the best use of time.

bart-v · 2022-03-11T08:34:56Z

We have been in touch with EMODnet Geology.
Their response is that it is not easily fixable, as the server is their main proxy for all of their websites.
So, they will need time to fix this.

annakrystalli · 2022-03-11T08:36:58Z

I think the simplest and most transparent option would be to just put a note in the README for Linux users, warning of the potential problem and explaining it is an issue with the affected servers. We can point them to the workaround but warn them about the security implications. Then it's really on EMODnet to ensure their servers are following best practice in security protocols which really is where the issue lies. I don't feel there is anything more we can or should do to handle it beyond what you've already done (I.e. notify of the issue).

maelle · 2022-03-11T09:16:44Z

@annakrystalli I'll close this PR, could you review the changes I made in #43?

annakrystalli added 3 commits February 5, 2022 13:20

do not update system reqs.

c841388

add libsodium-dev dependency for ows4r dep geometa

ce7f352

use apt-get instead of apt. More stable in script

ab1b593

test adding libcurl4-openssl-dev as linux dependency

a504f4a

maelle mentioned this pull request Mar 2, 2022

Change example in vignette as long as server is failing? #42

Closed

maelle closed this Mar 11, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix R-CMD-CHECK workflow ubuntu issues #39

Fix R-CMD-CHECK workflow ubuntu issues #39

annakrystalli commented Feb 5, 2022

codecov-commenter commented Feb 5, 2022 •

edited

annakrystalli commented Feb 5, 2022 •

edited

annakrystalli commented Feb 5, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022 •

edited

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

bart-v commented Mar 11, 2022

annakrystalli commented Mar 11, 2022

maelle commented Mar 11, 2022

Fix R-CMD-CHECK workflow ubuntu issues #39

Fix R-CMD-CHECK workflow ubuntu issues #39

Conversation

annakrystalli commented Feb 5, 2022

codecov-commenter commented Feb 5, 2022 • edited

Codecov Report

annakrystalli commented Feb 5, 2022 • edited

annakrystalli commented Feb 5, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022 • edited

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 1, 2022

maelle commented Mar 11, 2022

maelle commented Mar 11, 2022

bart-v commented Mar 11, 2022

annakrystalli commented Mar 11, 2022

maelle commented Mar 11, 2022

codecov-commenter commented Feb 5, 2022 •

edited

annakrystalli commented Feb 5, 2022 •

edited

maelle commented Mar 1, 2022 •

edited