Full-text is hard. Let's start with a linear search, and move towards something better.
total, n = 0, 0
results = 
while total < 100 and n < 1000:
for g in Group.objects.order_by('-last_seen')[n:100]:
if has_matching_query(g, query):
Another (working) implementation I did on this in another project: https://github.com/disqus/codebox/blob/master/codebox/apps/snippets/views.py#L31
Mostly as a prototype, but it should be fast enough
This has long been fixed.