Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

Close #116 - adding in a view to export user data as a CSV #119

Merged
merged 1 commit into from

3 participants

@rossbruniges

No description provided.

@alfredo alfredo commented on the diff
gameon/submissions/views.py
@@ -107,3 +109,41 @@ def single(request, slug, template='submissions/single.html'):
'entry': entry,
}
return render(request, template, data)
+
+
+def export_csv(request, template='submissions/table.html'):
+ if not request.user.is_staff:
+ return action_unavailable_response(request, case='not_staff')
+
+ response = HttpResponse(content_type='text/csv')
+ response['Content-Disposition'] = 'attachment; filename="gameon_entries.csv"'
@alfredo Owner
alfredo added a note

Maybe add a date and time to the file name to be able to differentiate between different exports?

Although this may not be that important.

Nice idea

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@alfredo
Owner

+1 looks good to me!

@alfredo alfredo commented on the diff
gameon/submissions/views.py
((14 lines not shown))
+
+ writer = csv.writer(response)
+ # Write out the header row
+ writer.writerow([
+ 'GAME NAME',
+ 'CATEGORY',
+ 'GAME WEBSITE',
+ 'GAME DESCRIPTION',
+ 'GAME SUBMISSION URL',
+ 'GAME VIDEO URL',
+ 'SUBMITTED BY',
+ 'SUBMITTED BIO',
+ 'SUBMIT TO MARKETPLACE?'
+ ])
+ for e in entry_set:
+ writer.writerow([
@alfredo Owner
alfredo added a note

As a word of advice, the writer doesn't support unicode by default. If unicode characters are expected on the title an different writer may need to be considered.

If the worst we get is a dodgy looking char here and there then I think that's fine. Is that what will happen or will the export fail?

@alfredo Owner
alfredo added a note

It would rise a 500 with a UnicodeEncoding error.

Hmm

How concerned are you with data integrity? If you don't mind losing the odd non-translatable character, you can encode it yourself before hand.

e.title.encode('ascii', 'ignore')

You can encode('ascii', 'replace') if you'd rather see ?s than nothing at all.

For extra Brownie points, you could normalise it before hand, which would go some way to dealing with characters that are ASCII-like (such as ä, é, etc).

import unicodedata
unicodedata.normalize('NFKD', e.title).encode('ascii', 'ignore')

@andrewhayward I've got a fix ready to land - just prepping the pull request, thanks for piling the feedback in!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@rossbruniges

Updated with a fix for potential unicode errors on export

@alfredo alfredo commented on the diff
gameon/base/utils.py
((6 lines not shown))
+# In case we get unicode in the DB we use a custom reader able to handle it
+class UTF8Recoder:
+ """
+ Iterator that reads an encoded stream and reencodes the input to UTF-8
+ """
+ def __init__(self, f, encoding):
+ self.reader = codecs.getreader(encoding)(f)
+
+ def __iter__(self):
+ return self
+
+ def next(self):
+ return self.reader.next().encode("utf-8")
+
+
+class UnicodeWriter:
@alfredo Owner
alfredo added a note

Lovely!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@alfredo
Owner

Looks awesome +1

@rossbruniges rossbruniges merged commit c6170bc into mozilla:master
@rossbruniges rossbruniges deleted the rossbruniges:data-export branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
3  gameon/base/templates/403.html
@@ -27,6 +27,9 @@ <h1 class="busta shout">Oops - looks like you've done something that you shouldn
<p>The Game On challenge hasn't opened yet, you can continue working on your game now but you won't be able to submit it until {{ request.challenge.start_date.strftime('%B %e, %Y') }}.</p>
{% endif %}
{% endif %}
+ {% if case == 'not_staff' %}
+ <p>Only members of the Mozilla Game On team are able to access this page.</p>
+ {% endif %}
</div>
<aside class="meta mini-col">
<h2 class="whimper">Here are some things you can do</h2>
View
48 gameon/base/utils.py
@@ -1,6 +1,9 @@
import datetime
import hashlib
import os
+import csv
+import codecs
+import cStringIO
from django.core.paginator import Paginator, InvalidPage, EmptyPage
from django.conf import settings
@@ -35,3 +38,48 @@ def get_paginator(queryset, page_number, items=settings.PAGINATOR_SIZE):
except (EmptyPage, InvalidPage):
paginated_query = paginator.page(paginator.num_pages)
return paginated_query
+
+
+# In case we get unicode in the DB we use a custom reader able to handle it
+class UTF8Recoder:
+ """
+ Iterator that reads an encoded stream and reencodes the input to UTF-8
+ """
+ def __init__(self, f, encoding):
+ self.reader = codecs.getreader(encoding)(f)
+
+ def __iter__(self):
+ return self
+
+ def next(self):
+ return self.reader.next().encode("utf-8")
+
+
+class UnicodeWriter:
@alfredo Owner
alfredo added a note

Lovely!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ """
+ A CSV writer which will write rows to CSV file "f",
+ which is encoded in the given encoding.
+ """
+
+ def __init__(self, f, dialect=csv.excel, encoding="utf-8", **kwds):
+ # Redirect output to a queue
+ self.queue = cStringIO.StringIO()
+ self.writer = csv.writer(self.queue, dialect=dialect, **kwds)
+ self.stream = f
+ self.encoder = codecs.getincrementalencoder(encoding)()
+
+ def writerow(self, row):
+ self.writer.writerow([s.encode("utf-8") for s in row])
+ # Fetch UTF-8 output from the queue ...
+ data = self.queue.getvalue()
+ data = data.decode("utf-8")
+ # ... and reencode it into the target encoding
+ data = self.encoder.encode(data)
+ # write to the target stream
+ self.stream.write(data)
+ # empty queue
+ self.queue.truncate(0)
+
+ def writerows(self, rows):
+ for row in rows:
+ self.writerow(row)
View
8 gameon/submissions/models.py
@@ -4,6 +4,7 @@
from django.db import models
from django.core.validators import MaxLengthValidator
from django.contrib.auth.models import AnonymousUser
+from django.core.urlresolvers import reverse
from django.conf import settings
from tower import ugettext_lazy as _
@@ -157,5 +158,12 @@ def get_entry_feature(self):
if self.thumbnail:
return '<img src="%s" alt=""/>' % self.get_image_src()
+ def get_absolute_url(self):
+ site_url = getattr(settings, 'SITE_URL', '')
+ entry_url = reverse('submissions.entry_single',
+ kwargs={'slug': self.slug})
+
+ return "%s%s" % (site_url, entry_url)
+
class Meta:
verbose_name_plural = 'Entries'
View
2  gameon/submissions/urls.py
@@ -5,6 +5,8 @@
urlpatterns = patterns('',
url(r'^apply/$', views.create, name='submissions.create_entry'),
url(r'^entries/all/$', views.list, name='submissions.entry_list'),
+ url(r'^entries/export/$', views.export_csv,
+ name='submissions.export_csv'),
url(r'^entries/in/(?P<category>[\w-]+)/$', views.list, name='submissions.entry_list'),
url(r'^entries/(?P<slug>[\w-]+)/$', views.single,
name='submissions.entry_single'),
View
43 gameon/submissions/views.py
@@ -2,14 +2,14 @@
from django.contrib import messages
from django.core.urlresolvers import reverse
from django.conf import settings
-from django.http import HttpResponseRedirect, Http404
+from django.http import HttpResponse, HttpResponseRedirect, Http404
from django.template.defaultfilters import slugify
from django.contrib.auth.decorators import login_required
from tower import ugettext as _
from gameon.base.views import action_unavailable_response
-from gameon.base.utils import get_page, get_paginator
+from gameon.base.utils import get_page, get_paginator, UnicodeWriter
from gameon.submissions.models import Entry, Category
from gameon.submissions.forms import EntryForm, NewEntryForm
@@ -107,3 +107,42 @@ def single(request, slug, template='submissions/single.html'):
'entry': entry,
}
return render(request, template, data)
+
+
+def export_csv(request, template='submissions/table.html'):
+ if not request.user.is_staff:
+ return action_unavailable_response(request, case='not_staff')
+
+ response = HttpResponse(content_type='text/csv')
+ response['Content-Disposition'] = 'attachment; filename="gameon_entries.csv"'
@alfredo Owner
alfredo added a note

Maybe add a date and time to the file name to be able to differentiate between different exports?

Although this may not be that important.

Nice idea

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+
+ entry_set = Entry.objects.all().order_by('-pk')
+
+ # using the custom Unicode writer so the view doesn't explode
+ writer = UnicodeWriter(response)
+ # Write out the header row
+ writer.writerow([
+ 'GAME NAME',
+ 'CATEGORY',
+ 'GAME WEBSITE',
+ 'GAME DESCRIPTION',
+ 'GAME SUBMISSION URL',
+ 'GAME VIDEO URL',
+ 'SUBMITTED BY',
+ 'SUBMITTED BIO',
+ 'SUBMIT TO MARKETPLACE?'
+ ])
+ for e in entry_set:
+ writer.writerow([
@alfredo Owner
alfredo added a note

As a word of advice, the writer doesn't support unicode by default. If unicode characters are expected on the title an different writer may need to be considered.

If the worst we get is a dodgy looking char here and there then I think that's fine. Is that what will happen or will the export fail?

@alfredo Owner
alfredo added a note

It would rise a 500 with a UnicodeEncoding error.

Hmm

How concerned are you with data integrity? If you don't mind losing the odd non-translatable character, you can encode it yourself before hand.

e.title.encode('ascii', 'ignore')

You can encode('ascii', 'replace') if you'd rather see ?s than nothing at all.

For extra Brownie points, you could normalise it before hand, which would go some way to dealing with characters that are ASCII-like (such as ä, é, etc).

import unicodedata
unicodedata.normalize('NFKD', e.title).encode('ascii', 'ignore')

@andrewhayward I've got a fix ready to land - just prepping the pull request, thanks for piling the feedback in!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ e.title,
+ e.category.name,
+ e.url,
+ e.description,
+ e.get_absolute_url(),
+ e.video_url,
+ e.created_by.display_name,
+ e.created_by.bio,
+ str(e.to_market),
+ ])
+
+ return response
Something went wrong with that request. Please try again.