Speed up the tag index page

[davidfischer]

Excluding spam projects from the tag index page really slows it down. This involves a project-project self join and then joining against users and the user profile. Each of these tables is a few 100k records. In the worst case, this page can take ~2s to load which can be very bad when the page is crawled repeatedly.

This PR simply removes filtering out spam projects which is the expensive part of this query.

• PRO: Speeds the page up drastically without getting rid of it. This page is one of the main ways for projects to get indexed by search engines. It also helps link projects to similar projects. In my tests, this sped things up by over an order of magnitude even in cases where the returned result didn't change from the spam filtering.
• CON: Spam projects might show up in the tag index. This probably wouldn't be noticed by users since spam projects don't have real tags that users might browse. However, it might help spam projects to get picked up by search engines although there are other ways search engines can find spam projects. This is mitigated significantly since we now have automated cleanup of spam projects. In total, there are ~2.7k spam projects that aren't autocleaned.

[humitos]

I'm fine with this solution. We are not doing a really good job by marking users/projects as spam anyways, so I'm sure that we are already showing spam projects on this listing currently.

Excluding projects from banned users is a nice to have that sometimes is taking our site down. At this moment, I prefer to show spam projects in that listing than taking the site down because of this.

Besides, if we do a better job by flagging users as spam, their projects will be automatically deleted by a task and they won't appear here anyways.

I may suggest to add a small comment about why we remove this filter, or even keep the line commented out together with the comment.

[ericholscher]

Yea, I think this is fine. We need to have better methods of making users as spam, or even denormalizing it onto the Project if we care about query speed.

Agreed with @humitos that it would be good to keep this line commented out explaining why it's been removed.

[davidfischer]

I re-added the code with a comment