Increase speed of fetching RSS feeds

[calebhearth]

Benchmarks for before and after the code change are available in individual
commits. This change results in an average time of 802604696.33ns for warm
requests, decreased from 1806248432ns. This is a 55.6% improvement.

[bernerdschaefer]

Though technically correct, its more common to call wg.Add(1) inside the for loop, just before the go func(...) call.

[calebhearth]

aight.

[bernerdschaefer]

Did you run this with -race (https://golang.org/doc/articles/race_detector.html)? There are a few things which don't seem likely to be safe with concurrency, like calling master.Add(item) from multiple goroutines.

[croaky]

Is -race another thing we can add to CI somewhere?

[calebhearth]

Did you run this with -race (https://golang.org/doc/articles/race_detector.html)? There are a few things which don't seem likely to be safe with concurrency, like calling master.Add(item) from multiple goroutines.

Totally right. I get WARNING: DATA RACE.

@bernerdschaefer pointed out that I should only be doing the fetching, not the merging into master. Since that'll require cross-thread communication I'll need to brush up on channel. Which I'll get to sometime next week probably.

[calebhearth]

And I'll be sure to add -race to the circle.yml

On Oct 2, 2015, at 4:48 PM, Dan Croak notifications@github.com wrote:

Is -race another thing we can add to CI somewhere?

—
Reply to this email directly or view it on GitHub.

[harlow]

Since the number of feeds is defined waitgroup may not be needed. Could just loop and grab from channel?

itemChan := make(chan *feeds.Item)

for _, feed := range sourceFeeds {
  go fetch(feed, itemChan)
}

for i := 1; i < len(sourceFeeds); i++ {  
  master.Add(<-itemChan)
}

And then thread the itemChan through the fetch instead of master

func fetch(feed sourceFeed, itemChan chan *feeds.Item) {

itemChan <- &feeds.Item{
  Title:       stripPodcastEpisodePrefix(items[i].Title),
  Link:        &feeds.Link{Href: items[i].Links[0].Href},
  Description: items[i].Description,
  Author:      &feeds.Author{Name: sourceName},
  Created:     published,
}

[croaky]

When I check out this branch and view it in browser, I don't see any feeds.