Speeding up indexing

[sorawee]

Just a FYI: indexing the whole catalog takes about 2 hours for me. But by adding concurrency, I can get it down to 30mins. It's unclear what's the cost/benefit. For me, who wants to generate pkgs-all.json.gz locally from scratch, the speedup is really helpful. But I also understand that there might be concerns about sending too many requests in a short period of time.

[samth]

I think adding more concurrency would be a good idea.