mx_api_content() fails if the last page doesn't contain any records

[mcguinlu]

This seems to be due to the fact that the number of records given by the "total" metadata is more than the total number of records actually available.

As of 14.39pm on 04/01/2021, the number of records given by the "total" is 148231. However, if you set the counter to any record within 31 of this figure (e.g. https://api.biorxiv.org/details/biorxiv/2013-01-01/2021-01-04/148201), you get a "No posts found" message. As medrxivr uses the "total" metadata field to work out how many pages it needs to cycle through to download the whole database, this sometimes leads to an error when the last page, expected by medrxivr based on the "total" field, is empty.

Note as more records are added to the API, the hardcode figures above will no longer demonstrate the issue.

[mcguinlu]

I think the best way to address this is to prevent the api_to_df() helper from failing if the cursor user counter in the for loop is the same as the maximum pages value

medrxivr/R/mx_api.R

Lines 80 to 96 in 83e686b

	for (cursor in 0:pages) {
	page <- cursor * 100
	page_link <- api_link(server,
	from_date,
	to_date,
	format(page,
	scientific = FALSE))
	tmp <- api_to_df(page_link)
	tmp <- tmp$collection
	df <- rbind(df, tmp)
	pb$tick(100)
	}

[mcguinlu]

Update: I've written to the CSHL team to flag this issue and they are looking into. In the meantime, the solution proposed above is probably the best bet.