Research performance impact of directing Sch A traffic to one replica

[lbeaufort]

We should research the performance impact of directing Sch A traffic to one replica and everything else to the other. The idea came from the notion of schedule A “hogging” resources from all the other queries. We could use the queries/activity levels from the outage drill to test before/after.

We probably want to look at RoutingSession

openFEC/webservices/common/models/base.py

Lines 8 to 47 in d111503

	class RoutingSession(SignallingSession):
	"""Route requests to database leader or follower as appropriate.
	Based on http://techspot.zzzeek.org/2012/01/11/django-style-database-routers-in-sqlalchemy/
	"""
	@property
	def followers(self):
	return self.app.config['SQLALCHEMY_FOLLOWERS']
	@property
	def follower_tasks(self):
	return self.app.config['SQLALCHEMY_FOLLOWER_TASKS']
	@property
	def restrict_follower_traffic_to_tasks(self):
	return self.app.config['SQLALCHEMY_RESTRICT_FOLLOWER_TRAFFIC_TO_TASKS']
	@property
	def use_follower(self):
	# Check for read operations and configured followers.
	use_follower = (
	not self._flushing and
	len(self.followers) > 0
	)
	# Optionally restrict traffic to followers for only supported tasks.
	if use_follower and self.restrict_follower_traffic_to_tasks:
	use_follower = (
	celery.current_task and
	celery.current_task.name in self.follower_tasks
	)
	return use_follower
	def get_bind(self, mapper=None, clause=None):
	if self.use_follower:
	return random.choice(self.followers)
	return super().get_bind(mapper=mapper, clause=clause)

This looks doable in theory: https://github.com/fecgov/openFEC/pull/4048/files
Need to set SQLA_FOLLOWERS env var to test

Completion criteria:

• Set up second replica on stage - asked Rohan about pairing on this around Wednesday Dec 11. Ideally we would put in a PR to the fec-infrastructure repo that would trigger the change.
• Set up additional application instances to match production
• Take a baseline locust test using research/locustfile from Research and prep for API outage drill #3965
• Push the test branch to stage that routes all schedule A traffic to one replica and everything else to another. Consider whether schedule A aggregates should get routed there too. https://github.com/fecgov/openFEC/tree/feature/test-sch-a-routing
  

  openFEC/webservices/common/models/base.py

  Lines 45 to 53 in 78d1701

  	if self.use_follower:
  	# TODO: Toggle to turn this on/off?
  	if len(self.followers) > 1:
  	if 'schedule_a' in request.path:
  	print("Schedule A in request path, using follower 1")
  	return self.followers[0]
  	else:
  	print("Schedule A not in request path, choosing another follower")
  	return random.choice(self.followers[1:])

• Take another locust test
• Synthesize results https://docs.google.com/spreadsheets/d/14IA_BOiIPKBLH6jN7LbI0s3FRZkPKOWsXfkqTPDUTKc/edit?folder=0APQeau5CCSjpUk9PVA#gid=0
• Put together a wiki on how to spin up/hook up another read replica in production: https://github.com/fecgov/openFEC/wiki/Creating-a-read-replica-in-Amazon-RDS-and-linking-it-to-the-API
• Remove 2nd replica from env vars, rebuild old release branch
• spin down 2nd replica (PR waiting for review)

Summary of results
Negative impact on Schedule A performance outweighs the improved performance of other endpoints. While this could be a stopgap measure to "quarantine" schedule A data, it's not a good long-term solution. Per @PaulClark2 I'm going to make a wiki with step-by-step instructions on spinning up another read replica in production, which would yield better results.