Unable to find documentation to set User Agent #22

pietjonker · 2021-07-05T09:31:45Z

When retrieving filings, I receive an error for being an Undeclared Automated Tool. For example, when using

latest_filings()

I receive error

No encoding supplied: defaulting to UTF-8.
Error in check_result(res) : 
  EDGAR request blocked from Undeclared Automated Tool.
Please visit https://www.sec.gov/developer for best practices.
See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
Consider also setting the environment variable 'EDGARWEBR_USER_AGENT

I found in the README the following information:

Because of abusive use of this library, the SEC is likely to block its use “as is” without setting a custom ‘User Agent’ identifier. Details for setting a custom agent are below.

However, no details were given below. Could anyone help me on setting the user agent?

The text was updated successfully, but these errors were encountered:

xbrl-data · 2021-07-21T03:20:30Z

Within edarWebR you need to set an env variable.
Sys.setenv('xxxx')

Then, i usually call
EDGARWEBR_USER_AGENT <- Sys.getenv('xxxx')

Also, for those of us that are getting errors due to the SEC shutting it down (despite never exceeding the stated limit on their webiste!), I've been substituting read_html with this:

html_retry <-  function(url1,   N.TRIES = 3L) {

  require(rvest)
  
  N.TRIES <- as.integer(N.TRIES)
  stopifnot(length(N.TRIES) == 1L, !is.na(N.TRIES))
  stopifnot(N.TRIES > 0L)
  
  while (N.TRIES > 0L) {
    result <- tryCatch(read_html(
      x = url1
    ), error = identity)
    if (!inherits(result, "error")) {
      break
    }
    ## Wait 0.5 seconds
    Sys.sleep(0.5)
    N.TRIES <- N.TRIES - 1L
  }
  
  if (N.TRIES == 0L) {
    stop(
      "'html_retry()' failed:",
      "\n  URL: ", url1,
      "\n  error: ", conditionMessage(result)
    )
  }
  
  invisible(result)
}

mirayoon · 2021-07-30T10:44:12Z

Hello,

thanks for developing this package. I set an env variable EDGARWEBR_USER_AGENT using Sys.setenv and tried 'filing_documents' function but I still receive the error message.

No encoding supplied: defaulting to UTF-8. Error in check_result(res) : EDGAR request blocked from Undeclared Automated Tool. Please visit https://www.sec.gov/developer for best practices. See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities Consider also setting the environment variable 'EDGARWEBR_USER_AGENT

could you help me how to proceed from here?

YuMan-Tam · 2021-07-31T22:21:34Z

@mirayoon : I have the same issue, but a try-catch block loop until successful like @xbrl-data mentioned will work.

mconsidine · 2021-09-29T20:34:05Z

Has anyone gotten this to work lately? I've been setting the user agent correctly per the SEC website and I'm still not able to return anything other than the messages above.
How has the try-catch routine as described been substituted for what xml2 has? Does all the underlying R code need to be editted?
Also, does Accept-Encoding: gzip, deflate need to somehow be set?
TIA,
mconsidine

xbrl-data · 2021-09-29T20:59:11Z

Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with.

mconsidine · 2021-09-30T01:07:09Z

I've managed to hack some of the code to work with a with_config wrapper around a GET call. But things only break down again later...On Sep 29, 2021 4:59 PM, xbrl-data ***@***.***> wrote: Honestly no; I've been basically locked out the last week or so from doing anything. My current workaround is to use docker and RSelenium, but I don't like that approach. Hopefully someone at the SEC fixes whatever they're messing with. —You are receiving this because you commented.Reply to this email directly, view it on GitHub, or unsubscribe.Triage notifications on the go with GitHub Mobile for iOS or Android.

mconsidine · 2021-09-30T02:09:51Z

It seems like some formatting at the SEC site has changed. E.g. the /feed/company-info node seems empty. Cik_ search works though.

Does anyone have a workflow they can share for low frequency requests for data access/parsing/report analysis if this is now busted? I was hoping to pair this with finstr-upd/xbrl-upd to show a college class.

Mconsidine

mconsidine · 2021-10-01T13:21:29Z

@xbrl-data : can you give me an idea how you have used those as a workaround? A PM would be fine, if you'd prefer.

IEORTools · 2022-01-19T23:36:09Z

@mirayoon I believe this is the proper way to set the environment variable

Sys.setenv(EDGARWEBR_USER_AGENT = " yourcompanyname youremail")

where yourcompanyname and youremail are specified by the user.

You can check it is properly set using the following

Sys.getenv("EDGARWEBR_USER_AGENT")

balthasars · 2022-01-23T17:42:41Z

@mirayoon I believe this is the proper way to set the environment variable

Sys.setenv(EDGARWEBR_USER_AGENT = " yourcompanyname youremail")

where yourcompanyname and youremail are specified by the user.

You can check it is properly set using the following

Sys.getenv("EDGARWEBR_USER_AGENT")

For me neitherSys.setenv() nor usethis::edit_r_environ() works — try using the fork I've suggested (see above).

granatakorps · 2022-04-02T18:42:15Z

Hi @balthasars thank you for sharing your solution but, unfortunately, it does not seem to work for me.
Any other tips on how to solve it?

mconsidine · 2022-04-04T12:52:37Z

This has worked for me:

library(tidyverse)
library(httr)
library(XBRL)
library(edgarWebR)

edgar_agent <- Sys.getenv("EDGARWEBR_USER_AGENT",
unset = "edgarWebR edgarpackage@examplemail.com"
)
ua <- httr::user_agent(edgar_agent)
options(HTTPUserAgent=ua$options$useragent)

Haven't tried to figure out if there's redundancy/unused vars being set in that ...

HTH,
mconsidine

snvv · 2022-05-25T18:12:49Z

Hello
None of the above approaches works for me

mconsidine · 2022-05-25T19:58:27Z

I used the version found here
https://github.com/balthasars/edgarWebR/tree/fix-user-agent
Also, this appeared to work, looking back over some code (thought I think its a kludge):
with_config(user_agent(ua$options$useragent),search.result <- xml2::read_html(url))
mconsidine

balthasars · 2022-05-25T21:04:42Z

Sorry for only getting back to this now, everyone!

So it appears that the install of my proposed fork doesn't work when installing it using remotes::install_github():

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/Rtmp2UYpBk/remotes894768da6952/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
#> No encoding supplied: defaulting to UTF-8.
#> Error in `mutate()`:
#> ! Problem while computing `results = purrr::map(search_terms,
#>   edgarWebR::company_search, type = "13F-HR")`.
#> Caused by error in `check_result()`:
#> ! EDGAR request blocked from Undeclared Automated Tool.
#> Please visit https://www.sec.gov/developer for best practices.
#> See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
#> Consider also setting the environment variable 'EDGARWEBR_USER_AGENT
bank_search_results
#> Error in eval(expr, envir, enclos): object 'bank_search_results' not found

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

^{Created on 2022-05-25 by the reprex package (v2.0.1)}

BUT: when I clone the repo, build the package in RStudio from there and then run my example, ta-dah:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

^{Created on 2022-05-25 by the reprex package (v2.0.1)}

I am puzzled by this behavior and will need to read up some more on environmental variables.

UPDATE:
The install using remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE) does also work if you set the environmental variable beforehand, using usethis::edit_r_environ(), such as EDGARWEBR_USER_AGENT = "Balthasar balthasar@balthasar.help:

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/RtmpvaBIOY/remotes937f2cfc9c69/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

# edgar_agent <- Sys.setenv(EDGARWEBR_USER_AGENT = "edgarWebR edgarpackage@examplemail.com")
# ua <- httr::user_agent(edgar_agent)
# options(HTTPUserAgent=ua$options$useragent)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] "Balthasar  balthasar@balthasar.help"

^{Created on 2022-05-25 by the reprex package (v2.0.1)}

So something must be overriding the correct value for the user agent variable....

mconsidine · 2022-05-25T22:23:32Z

I had been able to install and use the Balthazar version by specifying the @fix-user-agent branch in the "remotes" request. This was a few months ago and I'm not at my PC right now.

I'm not sure what if anything has changed lately, but a get company info request earlier today worked.

Mconsidine

twood886 · 2022-06-01T16:05:36Z

I had luck by changing the edgar_GET function

edgar_GET_V2 <- function(href, ua = httr::user_agent(Sys.getenv('EDGARWEBR_USER_AGENT'))){
res <- httr::GET(href, ua)
check_result(res)
return(res))

environment(edgar_GET_V2) <- asNamespace('edgarWebR')

assignInNamespace("edgar_GET", edgar_GET_V2, ns = "edgarWebR")

alejandroll10 · 2022-11-24T06:50:32Z

@twood886 solution worked for me

tripletvn · 2023-05-15T01:29:45Z

As of last week, @twood886 had the only solution that worked for me.
As of today, it no longer works.

spacecadet333 · 2023-08-30T17:04:16Z

Sorry for only getting back to this now, everyone!

So it appears that the install of my proposed fork doesn't work when installing it using remotes::install_github():

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/Rtmp2UYpBk/remotes894768da6952/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
#> No encoding supplied: defaulting to UTF-8.
#> Error in `mutate()`:
#> ! Problem while computing `results = purrr::map(search_terms,
#>   edgarWebR::company_search, type = "13F-HR")`.
#> Caused by error in `check_result()`:
#> ! EDGAR request blocked from Undeclared Automated Tool.
#> Please visit https://www.sec.gov/developer for best practices.
#> See https://mwaldstein.github.io/edgarWebR/index.html#ethical-use--fair-access for your responsibilities
#> Consider also setting the environment variable 'EDGARWEBR_USER_AGENT
bank_search_results
#> Error in eval(expr, envir, enclos): object 'bank_search_results' not found

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

Created on 2022-05-25 by the reprex package (v2.0.1)

BUT: when I clone the repo, build the package in RStudio from there and then run my example, ta-dah:

library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] ""

Created on 2022-05-25 by the reprex package (v2.0.1)

I am puzzled by this behavior and will need to read up some more on environmental variables.

UPDATE: The install using remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE) does also work if you set the environmental variable beforehand, using usethis::edit_r_environ(), such as EDGARWEBR_USER_AGENT = "Balthasar balthasar@balthasar.help:

remotes::install_github("https://github.com/balthasars/edgarWebR", force = TRUE)
#> Downloading GitHub repo balthasars/edgarWebR@HEAD
#> 
#> * checking for file ‘/private/var/folders/bm/3mv2bd6924gc4h_pz36ym8p80000gn/T/RtmpvaBIOY/remotes937f2cfc9c69/balthasars-edgarWebR-fb9a38e/DESCRIPTION’ ... OK
#> * preparing ‘edgarWebR’:
#> * checking DESCRIPTION meta-information ... OK
#> * checking for LF line-endings in source and make files and shell scripts
#> * checking for empty or unneeded directories
#> * building ‘edgarWebR_1.1.0.tar.gz’
library(dplyr)
#> 
#> Attaching package: 'dplyr'
#> The following objects are masked from 'package:stats':
#> 
#>     filter, lag
#> The following objects are masked from 'package:base':
#> 
#>     intersect, setdiff, setequal, union
library(tidyr)
library(edgarWebR)

# edgar_agent <- Sys.setenv(EDGARWEBR_USER_AGENT = "edgarWebR edgarpackage@examplemail.com")
# ua <- httr::user_agent(edgar_agent)
# options(HTTPUserAgent=ua$options$useragent)

bank_terms <- tibble::tribble(
   ~bank, ~search_terms,
   "BCV", "Banque Cantonale Vaudoise",
   "Credit Suisse", "Credit Suisse",
   "Edmond de Rotschild", c("Edmond de Rothschild"), 
   "Julius Baer", c("Julius Baer", "Bank Julius Baer"),
   "LGT", "LGT",
   "Lombard Odier", c("Lombard Odier","Compagnie Lombard Odier", "Compagnie Lombard"),
   "Pictet", c("Pictet", "Bank Pictet"),
   "UBS", c("UBS", "UBS AG"),
   "Union Bancaire Privée", c("UBP", "Union Bancaire Privee"),
   "Vontobel", "Vontobel",
   "ZKB", "Zurcher Kantonalbank"
   ) %>%
   unnest(search_terms)

bank_search_results <- bank_terms %>%
  mutate(
    results = purrr::map(
      search_terms,
      edgarWebR::company_search,
      type = "13F-HR"
    )
  ) %>% 
   unnest(results)
bank_search_results
#> # A tibble: 57 × 24
#>    bank              search_terms name  cik   fiscal_year_end company_href sic  
#>    <chr>             <chr>        <chr> <chr> <chr>           <chr>        <chr>
#>  1 BCV               Banque Cant… Banq… 0001… 1231            https://www… <NA> 
#>  2 Credit Suisse     Credit Suis… CRED… 0001… 1231            <NA>         <NA> 
#>  3 Credit Suisse     Credit Suis… CRED… 0000… <NA>            <NA>         <NA> 
#>  4 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         <NA> 
#>  5 Credit Suisse     Credit Suis… CRED… 0001… <NA>            <NA>         <NA> 
#>  6 Credit Suisse     Credit Suis… CRED… 0000… 1231            <NA>         6211 
#>  7 Edmond de Rotsch… Edmond de R… Edmo… 0001… 1231            <NA>         <NA> 
#>  8 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#>  9 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> 10 Edmond de Rotsch… Edmond de R… EDMO… 0001… 1231            <NA>         <NA> 
#> # … with 47 more rows, and 17 more variables: sic_description <chr>,
#> #   state_location <chr>, state_incorporation <chr>, mailing_city <chr>,
#> #   mailing_state <chr>, mailing_zip <chr>, mailing_street <chr>,
#> #   mailing_street2 <chr>, business_city <chr>, business_state <chr>,
#> #   business_zip <chr>, business_street <chr>, business_street2 <chr>,
#> #   business_phone <chr>, formerly <chr>, mailing_street1 <chr>,
#> #   business_street1 <chr>

Sys.getenv('EDGARWEBR_USER_AGENT')
#> [1] "Balthasar  balthasar@balthasar.help"

Created on 2022-05-25 by the reprex package (v2.0.1)

So something must be overriding the correct value for the user agent variable....

balthasars mentioned this issue Jan 23, 2022

very temporary fix of user agent problems in #22 #23

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Unable to find documentation to set User Agent #22

Unable to find documentation to set User Agent #22

pietjonker commented Jul 5, 2021

xbrl-data commented Jul 21, 2021 •

edited

Loading

mirayoon commented Jul 30, 2021

YuMan-Tam commented Jul 31, 2021

mconsidine commented Sep 29, 2021

xbrl-data commented Sep 29, 2021

mconsidine commented Sep 30, 2021 via email

mconsidine commented Sep 30, 2021

mconsidine commented Oct 1, 2021

IEORTools commented Jan 19, 2022 •

edited

Loading

balthasars commented Jan 23, 2022

granatakorps commented Apr 2, 2022

mconsidine commented Apr 4, 2022

snvv commented May 25, 2022

mconsidine commented May 25, 2022 •

edited

Loading

balthasars commented May 25, 2022 •

edited

Loading

mconsidine commented May 25, 2022 •

edited

Loading

twood886 commented Jun 1, 2022

alejandroll10 commented Nov 24, 2022

tripletvn commented May 15, 2023

spacecadet333 commented Aug 30, 2023

Unable to find documentation to set User Agent #22

Unable to find documentation to set User Agent #22

Comments

pietjonker commented Jul 5, 2021

xbrl-data commented Jul 21, 2021 • edited Loading

mirayoon commented Jul 30, 2021

YuMan-Tam commented Jul 31, 2021

mconsidine commented Sep 29, 2021

xbrl-data commented Sep 29, 2021

mconsidine commented Sep 30, 2021 via email

mconsidine commented Sep 30, 2021

mconsidine commented Oct 1, 2021

IEORTools commented Jan 19, 2022 • edited Loading

balthasars commented Jan 23, 2022

granatakorps commented Apr 2, 2022

mconsidine commented Apr 4, 2022

snvv commented May 25, 2022

mconsidine commented May 25, 2022 • edited Loading

balthasars commented May 25, 2022 • edited Loading

mconsidine commented May 25, 2022 • edited Loading

twood886 commented Jun 1, 2022

alejandroll10 commented Nov 24, 2022

tripletvn commented May 15, 2023

spacecadet333 commented Aug 30, 2023

xbrl-data commented Jul 21, 2021 •

edited

Loading

IEORTools commented Jan 19, 2022 •

edited

Loading

mconsidine commented May 25, 2022 •

edited

Loading

balthasars commented May 25, 2022 •

edited

Loading

mconsidine commented May 25, 2022 •

edited

Loading