Optimize repository scanning (GitHub API calls)

[hjacobs]

18:33:25.092 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Taking snapshot of organization 'zalando-stups'.
18:33:25.093 [http-nio-8080-exec-1] INFO  o.z.c.backend.scheduler.Fetcher - Submitted 1 TakeSnapshotTasks.
18:33:27.189 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Started collecting statistics for organization 'zalando-stups'.
18:33:35.223 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No contributors found for project 'stups-feedback' of organization 'zalando-stups'.
18:33:36.829 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No contributors found for project 'costreport' of organization 'zalando-stups'.
18:33:46.474 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Finished collecting statistics for organization 'zalando-stups'.
18:33:46.475 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Started collecting projects for organization 'zalando-stups'.
18:35:01.580 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No commits found for project 'stups-feedback' of organization 'zalando-stups'.
18:35:01.833 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No contributors found for project 'stups-feedback' of organization 'zalando-stups'.
18:35:12.587 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No commits found for project 'costreport' of organization 'zalando-stups'.
18:35:12.721 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No contributors found for project 'costreport' of organization 'zalando-stups'.
18:35:12.881 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Finished collecting projects for organization 'zalando-stups'.
18:35:12.881 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Started collecting contributors for organization 'zalando-stups'.
18:35:16.024 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No contributors found for project 'stups-feedback' of organization 'zalando-stups'.
18:35:16.160 [pool-3-thread-1] WARN  o.z.c.b.github.RepositoryWrapper - No contributors found for project 'costreport' of organization 'zalando-stups'.
18:35:25.062 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Finished collecting contributors for organization 'zalando-stups'.
18:35:25.062 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Started collecting languages for organization 'zalando-stups'.
18:35:27.504 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Finished collecting languages for organization 'zalando-stups'.
18:35:27.504 [pool-3-thread-1] INFO  o.z.c.b.github.TakeSnapshotTask - Successfully taken snapshot of organization 'zalando-stups'.
18:35:27.853 [http-nio-8080-exec-1] INFO  o.z.c.backend.scheduler.Fetcher - Successfully saved data for organization 'zalando-stups'.
18:35:27.853 [http-nio-8080-exec-1] INFO  o.z.c.backend.scheduler.Fetcher - Finished fetching data.

[LappleApple]

@hjacobs What's your suggestion for this one/ie how would you go about it?

[hjacobs]

@LappleApple that's not high priority, but we do some GitHub API calls twice --- i.e. taking longer than necessary (esp. if running in dev mode locally).

[Lauriapple1]

@hjacobs OK, we'll add to the stories list

[rbobin]

@hjacobs I couldn't see any duplicating API calls from the piece of log you provided..

[hjacobs]

You will see it if you look into the code, some aggregate counts basically do another call to the GitHub API again AFAIK.