Skip to content


Subversion checkout URL

You can clone with
Download ZIP


CSV export produces only one page #2449

joallard opened this Issue · 27 comments

Following a8ad4f9, CSVs can only be generated for the page, which is cumbersome if one has for instance 5000 records.

I've seen nothing in documentation about this. How do we get the whole dataset back?


Quoting from #2419:

The old behavior was: if you're on the first page, export 10,000 records. If you're on any other page, don't export anything.

The problem was that a true export functionality (where every single record is exported) wasn't ever fully implemented. Until such a PR exists, I'm removing the code that caused this, so CSV export respects pagination.

Also see #1924 for more related discussion.


The earliest ticket about this issue seems to be #346


I could see it making sense for those cases, they are surely admissible, but have you guys left a way to export all data/specify all dataset?

I'm wondering, is there already a setting to do this or easy config (doc issue); or would a proposal be helpful?


I'd very much like a PR addressing this issue. Currently there is no way to export everything. As I said in #1924:

Actually, a better option might be to make it a form that lets you insert both the offset and the number of records you want. Those fields would default to retrieve the same records you're currently viewing, but you could potentially set them to be anything.

As well, it would be a good idea to add a config option to limit the max number of records.

It'd also probably be a good idea to remove our custom CSV file creation code and use something that's either built-in to the current software stack (like we rely on for JSON & XML export) or that hooks into native DB features like postgres-copy does.


I came across a similar case where I wanted pagination, except for the CSV/JSON download links. I still needed to keep filtering/sorting/etc for the collection. It seemed simple enough to override the method that performs pagination on the collection if the request was in a format I didn't want paginated.

# Within app/admin/resource_name.rb
# Controller pagination overrides
controller do
    def apply_pagination(chain)
        chain = super unless formats.include?(:json) || formats.include?(:csv)

Is there a simpler way of doing this? Perhaps a configuration setting that allows tweaking pagination settings per format on a resource? By the way, thanks for all the work on this gem! It's saved huge amounts of time!


Nice workaround gsilis, I have just successfully used that override on one of my sites.

Personally I think downloading all records is most common use case of the CSV export and would like to see it being the default behaviour in active_admin. Or there could be an 'all' link next to the pagination, although displaying all records on the screen would be pretty slow for large data sets.


Just want to weigh in here agreeing that the default behavior should be to download the full dataset in CSV. On-screen admin display is for browsing/updating your records. Download is for crunching the data, and you need the full dataset for that. Just downgraded to 6.0 because of this issue.


@lauren CSV download never worked that way; it was limited to 10k records. I reverted that behavior because it's an unintuitive hack and we need a robust solution.


@Daxter 10K records at a time was definitely more useful than paginated downloads. Just sayin'!


I'd be incline to agree with @lauren's intuition, that's surely what I thought at first: download CSV would download all data. Nothing in the UI lets you believe there was a 10k limit.

Furthermore, I don't think that past behavior is the right guide for us. I think the right action would be to try to remove that limit and process all records. Maybe make that into an async process, kind of like Google Drive does when you download a group of large files zipped, but simpler.


I'm using the workaround from @gsilis for now and it works just fine. :+1: Not to go too far off topic but for large exports I often run into heroku's 30-second timeout limit, so i've been exploring pushing this into our sidekiq (async) queue. I don't think that functionality should necessarily be built into active admin though, as it seems like it would be too dependent on the server environment. Another quick fix we're exploring is to use streaming headers so we can start streaming the csv data directly to the browser as it's generated.


This is something that would be very useful for me as well.
I'm hoping to be able to get a PR together in the next couple of weeks to have a jQueryUI modal or something to specify from and to records.


+1 for a full export option

In my opinion, export links should obey filters but ignore pagination. Either that, or there needs to be a way to select all pages, similar to Django's admin tool.


Ok. So people don't seem to keen on a modal.

Something else I wondered about was having something in the DSL to allow the developer to specify what should happen when a large number of records are being exported.

For example, in one project I have (which doesn't use AA) when an export is requested, it's queued as a delayed job then emailed to the user.
Obviously, I'm not suggesting we tie people to using DJ, but give them the option of what to do in that case.

Something like

export do |collection, format|
  # All formatting, etc (all the things that take a long time) handled here,
  # probably by triggering something asynchronous.

  redirect_to :back, notice: "You will be emailed when your export is ready"

That's fine, but I still want a built-in solution so you don't have to have a job runner to be able to do large exports.


You can also set this as a before_filter:

before_filter only: :index do
  @per_page = 10_000 if request.format == 'text/csv'

This has now been reverted on 0-6-stable (37ba46c) and on master (bc791b3).


@seanlinsley though the commit has been reverted, the problem of exporting large resources is still there. Is there somewhere we can participate to finding a definitive solution? A new issue maybe?

@seanlinsley seanlinsley reopened this

Closing this was my mistake. This subject has caused me so much stress, I'd really like to resolve this soon.

#2626 exists as a solution, but has stalled. Is that somewhere near what you're imagining as a solution? Ultimately we need to ask the user how many records they want.


No worries @seanlinsley, it's not urgent (at least not for me)

Personally, I don't see the point of #2626. My preference would just be to find a way to export all records to CSV while keeping the request under 30 seconds.


That's the thing, "all records" could mean 5k or a million. How do we handle the huge data sets?


We have very large exports in our app. I use a combination of two solutions, one that streams the CSV response to the browser as it is generated (rather than try to stuff it all in a view and then deliver it at once) and another that queues a background job to generate the csv file and upload it to amazon s3. Depending on the type of export and how big they are I use one or the other.

For streaming:

For queueing background jobs with exports (we use sidekiq) that uploads the export to S3:


@stereoscott, thanks for csv streaming example, I'll definitely will try it


Thanks for the examples @stereoscott :heart:

@viniciusalmeida viniciusalmeida referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.

I can't get the streaming to work with or without comments. I either get an empty file or a file with "true". I prepared a Rails 4 update for months just to get to make this work, but I guess it's back to square one now.

Considering plataformatec/devise#2332 is closed for good, I have no idea how to export 20,000 records.


This has been resolved by #3038 :beers:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.