Extracting large datasets from SFDC reports

[StevenMMortimer]

Per @Met0o: What would you recommend as the optimal way to extract ~100k rows of data off SFDC reports using your Salesforcer package? Is that possible at all?

I am currently learning my way with the sf_run_report command but unless I am doing something wrong, I am getting only 2000 rows from the reports.

[StevenMMortimer]

The limitation is reported in the Salesforce API documentation:

The API returns up to the first 2,000 report rows. You can narrow results using filters.

^{https://developer.salesforce.com/docs/atlas.en-us.api_analytics.meta/api_analytics/sforce_analytics_rest_api_limits_limitations.htm}

Getting around this issue has been discussed in a few different forums with some of the following suggestions:

1. Replicate the logic of the report as a SOQL query which has no record limitation
2. Include at least 1 field on the record that can guarantee batches of no more than 2,000 records at a time and use that
   field as a filter in the report. Execute the report until you have paginated through all possible record values given that
   filter.

^{https://salesforce.stackexchange.com/questions/120236/how-can-i-get-a-report-to-return-more-than-2-000-rows-using-the-reportresults?rq=1}

I don't think I'll be able to create a function in the package that can implement either of these in a reliable way because they depend on specific knowledge of the Salesforce Org and the report. If you create a method to perform the pagination, please share back. I may add one myself in the future here on this issue or in a vignette.