Skip to content

kgoralski/microservice-production-readiness-checklist

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

16 Commits
 
 
 
 

Repository files navigation

Microservice Production Readiness Checklist

These are the principles that I have developed over the years of my professional career or by adapting the available knowledge in the literature to my work. I like to use these principles before deploying a new microservice (application) to production. I consider it also a good starting point for a discussion.

(* Yes, a lot of the points here can be covered via sane defaults provided by service templating)

Table of Contents

  1. The Checklist
  2. References
  3. Bonus

The Checklist

General Rules

Documentation

  • Readme.md - self-explanatory service name, how to run it locally and domain/sub-domain, bounded context described
  • Architecture docs / C4 Model diagrams
  • Service Catalog integration (e.g. Backstage)
  • API Open Specification file in root directory openapi.yaml
  • API versioning if needed

Testing and Quality

  • Linters (with reports that can be exported to e.g. SonarQube)
  • Automatic code Formatter (e.g. gofmt, ktfmt)
  • Test coverage above 70% (use common sense, just getting to the required number of coverage is not a goal here)
  • Functional/e2e/acceptance tests in place
  • Load Tests (at least basic ones) especially if higher traffic is expected
  • Contract Tests are recommended if there is service 2 service communication via HTTP (example: PACT tests)

Observability

  • Logging: All logs are written to STDOUT / STDERR. Logs are written in JSON. Configured verbosity levels. Correlation IDs, Mapped Diagnostic Context (MDC) is used. https://12factor.net/logs. Do not log any sensitive data. Shipped to e.g. ELK, Stackdriver, etc. Log levels like debug or trace should be not used by default.
  • Integration with a monitoring platforms. Dashboards in place. (e.g. NewRelic / Prometheus / Grafana)
  • Monitoring dashboards with Business Metrics (e.g. New Relic / Prometheus / Grafana)
  • Integration with a distributed tracing system: (e.g. New Relic)
  • Integration with error tracking system like (e.g. Sentry, New Relic)
  • Alerts are configured, there is Pager Duty integration in place and consider adding it to one of the escalation policies RunBooks (you may want to use Alert Manager integration)
  • Uptime, domain, SSL monitoring (optional) (e.g. StatusCake)
  • Real-time communication of application status (e.g. StatusPage)
  • Measure latency using percentiles. Discover the expected peak latency of the system, when healthy. Measure it and know 99%ile of the response times. Discover the expected peak throughput of my system, when healthy. Measure requests-per-minute or requests-per-second

Operations and Resiliency

  • There is staging environment or any other multi-tenancy environment - Don’t push directly to production
  • There is a rolling strategy defined
  • There is autoscaling in place (based on CPU, memory, traffic, events/messages e.g. HPA with K8S)
  • Graceful shutdown: The application understands SIGTERM and other signals and will gracefully shutdown itself after processing the current task. https://12factor.net/disposability
  • Configuration via environment: All important configuration options are read from the environment and the environment has higher priority over configuration files (but lower than the command line arguments). https://12factor.net/config
  • Health Checks: Readiness and Liveness probes
  • Resiliency, dependencies failure check, connection/socket timeouts, Retry Policies/circuit breakers in place. Safe to retry - idempotent calls etc. (The timeout for external service should be equal to 99%ile of this dependency's latency when healthy plus some breathing room). You can consider using service mesh.
  • Resiliency, If needed introduce the patterns like rate limiting and backpressure and/or load shedding to avoid overloading the system and making it unavailable for all users
  • Resiliency, implement fallback logic for every external dependency, for which this was possible and needed. Ask yourself “Is this resource needed to fulfill my business function, or can I (even partially) succeed without it?” (Remember that Fallbacks can be tricky and implementing a compensation job might be required)
  • Resiliency, consider using Cache. A cache provides an answer to the caller even if the underlying tiers fail to do so. A cache can also limit the need to make expensive calls to external services
  • Resource limits: Contains limits for memory, CPU, disk space, and any other available resources in the agreed format.
  • Artifact Management / Libraries & dependency proxy : JFROG Artifactory / Gitlab
  • Discovery / DNS configured
  • Setting DNS Cache TTL (hint for Java)
  • Dead letter queues or/and resistance to "bad" messages (if queues are used)
  • Feature Flags if needed (LaunchDarkly)
  • Locked versions of dependencies: Dependencies for package managers are fixed, including minor versions (For example, cool_framework = 2.5.3). Committed lock files are also a good way to do this.
  • Use private Docker Registry (AWS ECR , Gitlab)
  • Use Immutable deployments and if possible Canary deployments
  • When using containers like Docker - the only single process is running inside the container, with your application
  • Minimum 2 or 3 pods on production and/or PodDisruptionBudget/NodeAntiAffinity - whichever is appropriate to mitigate against a node failure - when running in Kubernetes
  • Define SLO/SLI/SLA
  • Build applications with Multi-tenancy in mind (sites, regions, users, etc.)

Database

  • Follow best practices from Cloud Provider (e.g. AWS Aurora) and prepare for Fast Failover
  • Data Org support (external team might have such requirements) - DB needs to be in provisioned mode instead of serverless if Bin Logs are used for exporting data, use bigger instances than t.small ones (e.g. t.small instances don’t support IAM access)
  • Database’s Connection string and Connection pool configured for the needed workload
  • Database (e.g. RDS - Aurora Writer/Reader endpoint consumed for better scalability, DocumentDB - use ReplicaSet in this case etc.)
  • Database integration with Backup Tooling - test regularly your restore process
  • Different Database users for RDS cluster admin and application usage
  • Maintenance window defined
  • Encryption enabled (e.g. with AWS KMS key)
  • Data Migration, migration scripts etc. (e.g. Flyway, Liquibase, gh-ost, Percona) - avoid manual changes on the database (tip: use a separate process for migrations ) - executed as separate process/job

Security and Compliance

  • If your service does not absolutely need to be directly accessible from the public Internet, do not expose it publicly - hide it behind VPN (at least)
  • If your service does need to be accessible through the public Internet
    • Authentication/Authorization in place if needed / JWT / Cognito / Auth0
    • Ensure it lives behind our Cloudfront CDN (and uses WAF if necessary)
  • Vulnerabilities scan check (e.g. SonarQube)
  • Run containers in Rootless mode
  • HTTPS (if needed)
  • Does not violate any licenses
  • GDPR data not exposed (https://gdpr-info.eu/art-4-gdpr/)
  • PII data not logged or stored without any good reason (ask your DPO) - Best practices to avoid sending Personally Identifiable Information (PII), Check Data Retention Policies
  • Configure bot to upgrade dependencies (e.g. Renovate bot)

Costs

  • Tags for Infrastructure / Terraform resources
  • Costs Dashboard for Weekly/Monthly costs (e.g. Cloudability)
  • Optimized CPU usage (eg. Reactive microservices)
  • Prepare Rightsizing plan

References

"Reviewing designs written by other engineers becomes an increasingly large (and important) part of our work life as we become more senior in our careers. We review designs for entire new systems written by partner developer teams. We review designs for pieces of automation to be developed and run by our own teams. Eventually we may find ourselves using review as a way to keep many teams in sync technically.

Most of us however, don't have a systematic way to approach reviews. We read the proposal or attend the meeting, and we look to our experience to try and predict problems. This is valuable, and experience can't be replaced—but I believe we can do better by applying both our expertise and a checklist of things to consider for each design."

Bonus

Because good design = Best Resilency.

Independent Service Heuristics - Team Topologies

Rules-of-thumb for identifying candidate value streams and domain boundaries by seeing if they could be run as a separate SaaS/cloud product.

  • Could it make any logical sense to offer this thing "as a service"?
  • Could you imagine this thing branded as a public cloud service (like AvocadoOnline.com )?
  • Could this thing be managed a viable cloud service in terms of revenue and customers?
  • Could the organisation currently track costs and investment in this thing separately from similar things?
  • Could this thing operate with minimal data from other sources?
  • Could this thing have a small/well-defined set of user types or customers (user personas)?
  • Could a team or set of teams effectively build and operate a service based on this thing?

About

The principles that help to deploy safely to the production environment. If you like it:

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published