Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in http_statuses[[as.character(status)]] : subscript out of bounds #567

Closed
Javdat opened this issue Jan 21, 2019 · 2 comments

Comments

@Javdat
Copy link
Contributor

commented Jan 21, 2019

Sometimes I am getting an error Error in http_statuses[[as.character(status)]] : subscript out of bounds while calling s3saveRDS() in aws.s3. As far as I could pinpoint, the issue originates from httr, more specifically in http_status.

This is the part of the code that is (I think) is problematic:

  status <- status_code(x)

  status_desc <- http_statuses[[as.character(status)]]
  if (is.na(status_desc)) {
    stop("Unknown http status code: ", status, call. = FALSE)
  }

This assumes that http_statuses[[as.character(status)]] will return NA when it is faced with a status code not present in http_statuses, and thus will be handed with: "Unknown http status code: ".

However, in reality, it seems http_statuses[[as.character(status)]] will just throw subscript out of bounds error.

Reproducible(ish) example:

Status present in http_statuses, works as expected:

> status <- status_code(100)
> status_desc <- httr:::http_statuses[[as.character(status)]]
> status_desc
[1] "Continue"

Unknown status, fails before reaching "Unknown http status code: " handler:

> status <- status_code(1000)
> status_desc <- httr:::http_statuses[[as.character(status)]]
Error in httr:::http_statuses[[as.character(status)]] : 
  subscript out of bounds
> status_desc
Error: object 'status_desc' not found

So, I think either httr:::http_statuses[[as.character(status)]] has to be called safely and error handled as NA, or indexing should happen one level only.

> status <- status_code(1000)
> status_desc <- httr:::http_statuses[as.character(status)] # Notice [] instead of [[]]
> status_desc
<NA> 
  NA 
> as.character(status_desc) # remove variable name
[1] NA

@Javdat Javdat changed the title Error in httr:::http_statuses[[as.character(status)]] : subscript out of bounds Error in http_statuses[[as.character(status)]] : subscript out of bounds Jan 21, 2019

@sheffe

This comment has been minimized.

Copy link

commented Feb 25, 2019

Found this existing issue after lots of similar error messages. For what it's worth, I make ~100k read/write calls through aws.s3 every week, and I see this error on something like 1 in 200 of those calls. I've done some limited diagnostic checks to see whether this error happens more often on specific types of calls, payload sizes, etc., but it seems pretty random.

(I haven't been able to create a full reprex without doing something objectively dumb, like "write 100k tiny RDS files with aws.s3::s3saveRDS and see what happens." The example above is a nice demo of what happens.)

From my vantage point, this low-level error from httr is a complication because most of the aws.s3 high-level functions have been difficult to wrap in purrr adverbs like safely/slowly/insistently. By that I mean errors like this inside a package wrapper function can still cause a hard interruption, for reasons I haven't been able to trace yet. Any research to that end probably belongs in aws.s3 directly. I don't use many other high-level API packages that wrap httr and have this kind of call volume, but I suspect the general problem won't be exclusive to dealing with AWS resources.

Both httr and aws.s3 have large enough user footprints that I'd appreciate a maintainer's opinion on where to submit a PR -- I think @Javdat 's solution above makes sense, but after some digging I can't trace what else would be impacted.

(PS: For my use-cases, 100k read/writes a week with only a 0.5% error rate is pretty remarkable. Thanks for httr!)

@hadley hadley added the bug label Apr 1, 2019

@hadley

This comment has been minimized.

Copy link
Member

commented Apr 1, 2019

I think is a fairly simple fix if someone wants to do a PR. Just check before subsetting:

  if (!http_statuses %in% http_statuses) {
    stop("Unknown http status code: ", status, call. = FALSE)
  }
  status_desc <- http_statuses[[as.character(status)]]

Javdat pushed a commit to Javdat/httr that referenced this issue Apr 8, 2019

Javdat pushed a commit to Javdat/httr that referenced this issue Apr 10, 2019

Javdat pushed a commit to Javdat/httr that referenced this issue Apr 11, 2019

hadley added a commit that referenced this issue Apr 28, 2019

@hadley hadley closed this Apr 28, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.