submit_form fails to submit OpenReview login form

[kevinrue]

I've been trying to log in to https://openreview.net/ using rvest, following instructions at https://riptutorial.com/r/example/23955/using-rvest-when-login-is-required and https://stackoverflow.com/questions/54132813/rvest-how-to-submit-form-when-input-doesnt-have-a-name

library(rvest)

# Address of the login webpage
login <- "https://openreview.net/login"

# Create a web session with the desired login address
pgsession <- html_session(login)
pgform <- html_form(pgsession)[[2]]  #in this case the submit is the 2nd form
pgform$fields[[1]]$value <- "****" # replace by email
pgform$fields[[2]]$value <- "***" # replace by password
submit_form(pgsession, pgform)

which results in

Submitting with '<unnamed>'
Error in parse_url(url) : length(url) == 1 is not TRUE

Reading the submit_form function, I noticed that it builds a URL as follows

url <- xml2::url_absolute(form$url, session$url)

Checking the form, I noticed that form$url is NULL, which results in url being NULL, explaining the message length(url) == 1 is not TRUE.

I've tried fiddling with the rvest source code, e.g. editing lines to say

form_url <- ifelse(!is.null(form$url), "/", form$url)
  url <- xml2::url_absolute(form_url, session$url)

but the reprex above then produces

Submitting with '<unnamed>'
Error in ans[npos] <- rep(no, length.out = len)[npos] : 
  replacement has length zero
In addition: Warning message:
In rep(no, length.out = len) : 'x' is NULL so the result will be NULL

I'm all out of ideas, whether the form doesn't respect standards that rvest expects, or rvest needs some further tweaks to support whatever this form expects./returns.

Thanks in advance for the advice!

[hadley]

Looks like this login form requires js but I think rvest could produce a better error when url is NULL.

[kevinrue]

Thanks @hadley

Having contacted the OpenReview team, they indicated that there is a Python API to interact with their platform.

You may need to use the API (api.openreview.net) to login in the system. Please take a look at your python client for reference: https://github.com/openreview/openreview-py/blob/master/openreview/openreview.py#L115

Also you have documentation about the API here: https://api.openreview.net/api/#/Users/login

I've started (locally, not on github yet) an R package that uses basilisk / reticulate to wrap Python, but ultimately, I think I'll just write a bit of python ;)

Happy to close the issue, or to leave it open as a sticky note if you want to work on any related aspect of rvest

[hadley]

I added a better error in 522f930