Skip to content
These notes describes how to improve NGINX performance, security and other important things; @ssllabs A+ 100%, @mozilla A+ 120/100.
Branch: master
Clone or download
trimstray add '--depth 1' to git clone commands
- signed-off-by: trimstray <trimstray@gmail.com>
Latest commit 6b7a31c May 14, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
lib/nginx minor fixes; separation of installation methods May 13, 2019
static/img change repository main logo May 8, 2019
CODE_OF_CONDUCT.md init commit Jan 27, 2019
CONTRIBUTING.md fixed typos Mar 1, 2019
LICENSE.md init commit Jan 27, 2019
README.md add '--depth 1' to git clone commands May 14, 2019

README.md

Nginx Admin's Handbook

My notes on NGINX administration basics, tips & tricks, caveats, and gotchas.

Meme


Hi-diddle-diddle, he played on his
fiddle and danced with lady pigs.
Number three said, "Nicks on tricks!
I'll build my house with EN-jin-EKS!".
The Three Little Pigs: Who's Afraid of the Big Bad Wolf?


Pull Requests License

Created by trimstray and contributors


Table of Contents

Introduction



Before using the NGINX please read Beginner’s Guide. It's a great introduction for everyone.

Nginx (/ˌɛndʒɪnˈɛks/ EN-jin-EKS, stylized as NGINX or nginx) is an HTTP and reverse proxy server, a mail proxy server, and a generic TCP/UDP proxy server. It is originally written by Igor Sysoev. For a long time, it has been running on many heavily loaded Russian sites including Yandex, Mail.Ru, VK, and Rambler.

NGINX is a fast, light-weight and powerful web server that can also be used as a load balancer and caching server. It provides the core of complete web stacks.

To increase your knowledge, read Getting Started and NGINX Documentation resources.

General disclaimer

It is not official document. This is rather a collection of rules, notes and papers, best practices and recommendations used by me (also in production environments but not only). Many of these refer to external resources.

I've never found one guide that covers the most important things about NGINX, and around NGINX. Of course, we have official documentation - it's probably the best place for us.

I think, however, there hasn't been a truly in-depth cheatsheet which describe a variety of configurations and important cross-cutting topics for HTTP servers. That's why I created this repository to help us to configure high performing NGINX web and proxy servers that are fast, secure and stable. I still have a lot to improve and to do.

Throughout this handbook you will explore the many features of NGINX and how to use them. This guide is fairly comprehensive, and touches a lot of the functions (e.g. security, performance) of NGINX.

If you do not have the time to read hundreds of articles this multipurpose handbook may be useful. I hope you enjoy it.

Before you start remember about the two most important things:

Do not follow guides just to get 100% of something. Think about what you actually do at your server!

These guidelines provides recommendations for very restrictive setup.

Contributing & Support

If you find something which doesn't make sense, or something doesn't seem right, please make a pull request and please add valid and well-reasoned explanations about your changes or comments.

Before adding a pull request, please see the contributing guidelines.

If this project is useful and important for you, you can bring positive energy by giving some good words or supporting this project. Thank you!

ToDo list

New chapters:

  • Caching
  • 3rd party modules
  • Web Application Firewall
  • ModSecurity

Existing chapters:

Books
  • ModSecurity 3.0 and NGINX: Quick Start Guide
  • Cisco ACE to NGINX: Migration Guide
External Resources
  • Nginx official
    • Nginx Official Forum
    • Nginx Official Mailing List
  • Static analyzers
    • nginx-minify-conf
Helpers
  • Configuration syntax
    • Enable syntax highlight for Nginx configuration file
    • Measurement units
    • Comments
    • Variables & Strings
    • Directives, Blocks, and Contexts
  • Debugging
    • Check that the gzip_static module is working
    • Which worker processing current request
  • Configuration snippets
    • Custom error pages
    • Adding and removing the www prefix
    • Rewrite POST request with payload to external endpoint
    • Allow multiple cross-domains using the CORS headers
    • Tips and methods for high load traffic testing (cheatsheet)
  • Installation from source
    • Add SystemTap - Real-time analysis and diagnoistcs tools
    • Separation and improvement of installation methods
    • Add installation process on FreeBSD 11.2 (separate file)
    • Add installation process on CentOS 7 for Tengine Web Server (separate file)
Base Rules
  • Format, prettify and indent your Nginx code
  • Never use a hostname in a listen directive
  • Making a rewrite absolute (with scheme)
  • Use "return" directive for URL redirection (301, 302)
  • Use mirror module to copy requests to another backend
Performance
  • Use "index" directive in the http block
  • Avoid multiple "index" directives
  • Use "$request_uri" to avoid using regular expressions
  • Use "try_files" directive to ensure a file exists
  • Don't pass all requests to backends - use "try_files"
  • Set proxy timeouts for normal load and under heavy load
  • Configure kernel parameters for high load traffic
Hardening
  • Use only the latest supported OpenSSL version
  • Set properly files and directories permissions (also with acls) on a paths
  • Implement HTTPOnly and secure attributes on cookies
Others
  • Define security policies with security.txt

Reports: blkcipher.info

Many of these recipes have been applied to the configuration of my private website.

An example configuration is in configuration examples chapter. It's also based on this version of printable high-res hardening checklist.

SSL Labs

Read about SSL Labs grading here (SSL Labs Grading 2018).

A+ is clearly the desired grade, both A and B grades are acceptable and result in adequate commercial security. The B grade, in particular, may be applied to configurations designed to support very wide audiences (for old clients).

I finally got A+ grade and following scores:

  • Certificate = 100%
  • Protocol Support = 100%
  • Key Exchange = 90%
  • Cipher Strength = 90%

blkcipher_ssllabs_preview

Mozilla Observatory

Read about Mozilla Observatory here.

I also got the highest note from Mozilla:

blkcipher_mozilla_observatory_preview

Printable high-res hardening checklists

I created printable posters with hardening checklists (High-Res 5000x8200) based on these recipes:

For *.xcf and *.pdf formats please see this directory.

  • A+ with all 100%’s on @ssllabs and 120/100 on @mozilla observatory:

    It provides the highest scores of the SSL Labs test. Setup is very restrictive with 4096-bit private key, only TLS 1.2 and also modern strict TLS cipher suites (non 128-bits).

nginx-hardening-checklist-100p

  • A+ on @ssllabs and 120/100 on @mozilla observatory with TLS 1.3 support:

    It provides less restrictive setup with 2048-bit private key, TLS 1.2 and 1.3 and also modern strict TLS cipher suites (128/256-bits). The final grade is also in line with the industry standards. Recommend using this configuration.

nginx-hardening-checklist-tls13

Books

These books are probably pay or free. They can be official and unofficial.

Nginx Essentials

Authors: Valery Kholodkov

Excel in Nginx quickly by learning to use its most essential features in real-life applications.

  • Learn how to set up, configure, and operate an Nginx installation for day-to-day use
  • Explore the vast features of Nginx to manage it like a pro, and use them successfully to run your website
  • Example-based guide to get the best out of Nginx to reduce resource usage footprint

This short review comes from this book or the store.

Nginx Cookbook

Authors: Derek DeJonghe

You’ll find recipes for:

  • Traffic management and A/B testing
  • Managing programmability and automation with dynamic templating and the NGINX Plus API
  • Securing access through encrypted traffic, secure links, HTTP authentication subrequests, and more
  • Deploying NGINX to AWS, Azure, and Google cloud-computing services
  • Using Docker to deploy containers and microservices
  • Debugging and troubleshooting, performance tuning, and practical ops tips

This short review comes from this book or the store.

Nginx HTTP Server

Authors: Martin Fjordvald, Clement Nedelcu

Harness the power of Nginx to make the most of your infrastructure and serve pages faster than ever.

  • Discover possible interactions between Nginx and Apache to get the best of both worlds
  • Learn to exploit the features offered by Nginx for your web applications
  • Get your hands on the most updated version of Nginx (1.13.2) to support all your web administration requirements

This short review comes from this book or the store.

Nginx High Performance

Authors: Rahul Sharma

Optimize NGINX for high-performance, scalable web applications.

  • Configure Nginx for best performance, with configuration examples and explanations
  • Step–by-step tutorials for performance testing using open source software
  • Tune the TCP stack to make the most of the available infrastructure

This short review comes from this book or the store.

Mastering Nginx

Authors: Dimitri Aivaliotis

Written for experienced systems administrators and engineers, this book teaches you from scratch how to configure Nginx for any situation. Step-by-step instructions and real-world code snippets clarify even the most complex areas.

This short review comes from this book or the store.

ModSecurity 3.0 and NGINX: Quick Start Guide

Authors: Faisal Memon, Owen Garrett, Michael Pleshakov

Learn in this ebook how to get started with ModSecurity, the world’s most widely deployed web application firewall (WAF), now available for NGINX and NGINX Plus.

This short review comes from this book or the store.

Cisco ACE to NGINX: Migration Guide

Authors: Faisal Memon

This ebook provides step-by-step instructions on replacing Cisco ACE with NGINX and off-the-shelf servers. NGINX helps you cut costs and modernize.

In this ebook you will learn:

  • How to migrate Cisco ACE configuration to NGINX, with detailed examples
  • Why you should go with a software load balancer, and not hardware

This short review comes from this book or the store.

External Resources

Nginx official

  ▪️ Nginx Project
  ▪️ Nginx Documentation
  ▪️ Nginx Wiki
  ▪️ Nginx Admin's Guide
  ▪️ Nginx Pitfalls and Common Mistakes
  ▪️ Nginx Forum
  ▪️ Nginx Mailing List
  ▪️ Nginx Read-only Mirror

Based on the Nginx

  ▪️ OpenResty
  ▪️ The Tengine Web Server

Cheatsheets & References

  ▪️ Nginx Cheatsheet
  ▪️ Nginx Tutorials, Linux Sysadmin Configuration & Optimizing Tips and Tricks
  ▪️ Nginx boilerplate configs
  ▪️ Awesome Nginx configuration template
  ▪️ Nginx Quick Reference
  ▪️ A collection of resources covering Nginx and more
  ▪️ A collection of useful Nginx configuration snippets
  ▪️ agentzh's Nginx Tutorials

Performance & Hardening

  ▪️ Nginx Tuning For Best Performance by Denji
  ▪️ TLS has exactly one performance problem: it is not used widely enough
  ▪️ SSL/TLS Deployment Best Practices
  ▪️ SSL Server Rating Guide
  ▪️ How to Build a Tough NGINX Server in 15 Steps
  ▪️ Top 25 Nginx Web Server Best Security Practices
  ▪️ Nginx Secure Web Server
  ▪️ Strong ciphers for Apache, Nginx, Lighttpd and more
  ▪️ Strong SSL Security on Nginx
  ▪️ Enable cross-origin resource sharing (CORS)
  ▪️ NAXSI - WAF for Nginx
  ▪️ ModSecurity for Nginx
  ▪️ Transport Layer Protection Cheat Sheet
  ▪️ Security/Server Side TLS

Playgrounds

  ▪️ NGINX Rate Limit, Burst and nodelay sandbox

Config generators

  ▪️ Nginx config generator on steroids
  ▪️ Mozilla SSL Configuration Generator

Static analyzers

  ▪️ gixy - is a tool to analyze Nginx configuration to prevent security misconfiguration and automate flaw detection.
  ▪️ nginx-config-formatter - Nginx config file formatter/beautifier written in Python.
  ▪️ nginxbeautifier - format and beautify nginx config files.
  ▪️ nginx-minify-conf - creates a minified version of a nginx configuration.

Log analyzers

  ▪️ GoAccess - is a fast, terminal-based log analyzer (quickly analyze and view web server statistics in real time).
  ▪️ Graylog - is a leading centralized log management for capturing, storing, and enabling real-time analysis.
  ▪️ Logstash - is an open source, server-side data processing pipeline.

Performance analyzers

  ▪️ ngxtop - parses your nginx access log and outputs useful, top-like, metrics of your nginx server.

Benchmarking tools

  ▪️ ab - is a single-threaded command line tool for measuring the performance of HTTP web servers.
  ▪️ siege - is an http load testing and benchmarking utility.
  ▪️ wrk - is a modern HTTP benchmarking tool capable of generating significant load.
  ▪️ bombardier - is a HTTP(S) benchmarking tool.
  ▪️ gobench - is a HTTP/HTTPS load testing and benchmarking tool.
  ▪️ hey - is a HTTP load generator, ApacheBench (ab) replacement, formerly known as rakyll/boom.
  ▪️ boom - is a script you can use to quickly smoke-test your web app deployment.
  ▪️ JMeter™ - is designed to load test functional behavior and measure performance.
  ▪️ Gatling - is a powerful open-source load and performance testing tool for web applications.
  ▪️ locust - is an easy-to-use, distributed, user load testing tool.

Development

  ▪️ Emiller’s Guide To Nginx Module Development
  ▪️ An Introduction To OpenResty (nginx + lua) - Part 1
  ▪️ An Introduction To OpenResty - Part 2 - Concepts
  ▪️ An Introduction To OpenResty - Part 3

Online tools

  ▪️ SSL Server Test by SSL Labs
  ▪️ SSL/TLS Capabilities of Your Browser
  ▪️ Test SSL/TLS (PCI DSS, HIPAA and NIST)
  ▪️ SSL analyzer and certificate checker
  ▪️ Test your TLS server configuration (e.g. ciphers)
  ▪️ Scan your website for non-secure content
  ▪️ Public Diffie-Hellman Parameter Service/Tool
  ▪️ Analyse the HTTP response headers by Security Headers
  ▪️ Analyze your website by Mozilla Observatory
  ▪️ CAA Record Helper
  ▪️ Linting tool that will help you with your site's accessibility, speed, security and more
  ▪️ Service to scan and analyse websites
  ▪️ Tool from above to either encode or decode a string of text
  ▪️ Online translator for search queries on log data
  ▪️ Online regex tester and debugger: PHP, PCRE, Python, Golang and JavaScript
  ▪️ Online tool to learn, build, & test Regular Expressions
  ▪️ Online Regex Tester & Debugger
  ▪️ A web app for encryption, encoding, compression and data analysis
  ▪️ Nginx location match tester
  ▪️ Nginx location match visible

Other stuff

  ▪️ Transport Layer Security (TLS) Parameters
  ▪️ Security/Server Side TLS by Mozilla
  ▪️ TLS Security 6: Examples of TLS Vulnerabilities and Attacks
  ▪️ Guidelines for Setting Security Headers
  ▪️ Secure your web application with these HTTP headers
  ▪️ Security HTTP Headers
  ▪️ Analysis of various reverse proxies, cache proxies, load balancers, etc.
  ▪️ TLS Redirection (and Virtual Host Confusion)
  ▪️ HTTPS on Stack Overflow: The End of a Long Road
  ▪️ The Architecture of Open Source Applications - Nginx
  ▪️ BBC Digital Media Distribution: How we improved throughput by 4x
  ▪️ The C10K problem by Dan Kegel
  ▪️ Web cache server performance benchmark: nuster vs nginx vs varnish vs squid
  ▪️ High Performance Browser Networking
  ▪️ jemalloc vs tcmalloc vs dlmalloc

Helpers

Nginx directories and files

If you compile NGINX server by default all files and directories are available from /usr/local/nginx location.

For prebuilt NGINX package paths can be as follows:

  • /etc/nginx - is the default configuration root for the NGINX service
  • /etc/nginx/nginx.conf - is the default configuration entry point used by the NGINX services. Includes the top-level http block and all other configuration files
  • /usr/share/nginx - is the default root directory for requests, contains html directory and basic static files
  • /var/log/nginx - is the default log (access and error log) location for NGINX
  • /var/lib/nginx or /var/cache/nginx - is the default temporary files location for NGINX
  • /etc/nginx/conf, /etc/nginx/conf.d or /etc/nginx/sites-enabled - contains custom/vhosts configuration files
  • /usr/local/nginx/logs or /var/run/nginx - contains information about NGINX process(es)

Nginx commands

  • nginx -h - shows the help
  • nginx -v - shows the NGINX version
  • nginx -V - shows the extended information about NGINX: version, build parameters and configuration arguments
  • nginx -t - tests the NGINX configuration
  • nginx -c - sets configuration file (default: /etc/nginx/nginx.conf)
  • nginx -p - sets prefix path (default: /etc/nginx/)
  • nginx -T - tests the NGINX configuration and prints the validated configuration on the screen
  • nginx -s - sends a signal to the NGINX master process:
    • stop - discontinues the NGINX process immediately
    • quit - stops the NGINX process after it finishes processing inflight requests
    • reload - reloads the configuration without stopping NGINX processes
    • reopen - instructs NGINX to reopen log files
  • nginx -g - sets global directives out of configuration file

Configuration syntax

Comments

NGINX configuration files don't support comment blocks; they only accept # at the beginning of a line for a comment.

End of lines

Lines containing directives must end with a ; or NGINX will fail to load the configuration and report an error.

Variables & Strings

Variables in NGINX start with $. Some modules introduce variables can be used when setting directives.

There are some directives that do not support variables, e.g. access_log, or error_log.

Strings may be inputted without quotes unless they include blank spaces, semicolons or curly braces, then they need to be escaped with backslashes or enclosed in single/double quotes.

Variables in quoted strings are expanded normally unless the $ is escaped.

Directives, Blocks, and Contexts

Read this great article about the NGINX configuration inheritance model by Martin Fjordvald.

Configuration options in NGINX are called directives. We have four types of directives in NGINX:

  • standard directive - one value per context, for example:
    worker_connections 512;
  • array directive - multiple values per context, for example:
    error_log /var/log/nginx/localhost/localhost-error.log warn;
  • action directive - something which does not just configure, for example:
    rewrite ^(.*)$ /msie/$1 break;
  • try_files directive, for example:
try_files $uri $uri/ /test/index.html;

Directives are organized into groups known as blocks or contexts. It appears to be organized in a tree-like structure, defined by sets of brackets - { and }. It's a simple structure and very transparent.

As a general rule, if a directive is valid in multiple nested scopes, a declaration in a broader context will be passed on to any child contexts as default values.

The NGINX contexts can be layered within one another so NGINX provides a level of inheritance. The NGINX contexts structure looks like this:

Global/Main Context
        |
        |
        +-----» Events Context
        |
        |
        +-----» HTTP Context
        |          |
        |          |
        |          +-----» Server Context
        |          |          |
        |          |          |
        |          |          +-----» Location Context
        |          |
        |          |
        |          +-----» Upstream Context
        |
        |
        +-----» Mail Context
Measurement units

Sizes can be specified in:

  • k or K: Kilobytes
  • m or M: Megabytes
  • g or G: Gigabytes
client_max_body_size 2M;

Time intervals can be specified in:

  • ms: Milliseconds
  • s: Seconds (default, without a suffix)
  • m: Minutes
  • h: Hours
  • d: Days
  • w: Weeks
  • M: Months (30 days)
  • y: Years (365 days)
proxy_read_timeout 20s;
Enable syntax highlight for Nginx configuration file
vi/vim
# 1) Download vim plugin for NGINX:

# Official NGINX vim plugin:
mkdir -p ~/.vim/syntax/

wget "http://www.vim.org/scripts/download_script.php?src_id=19394" -O ~/.vim/syntax/nginx.vim

# Improved NGINX vim plugin (incl. syntax highlighting) with Pathogen:
mkdir -p ~/.vim/{autoload,bundle}/

curl -LSso ~/.vim/autoload/pathogen.vim https://tpo.pe/pathogen.vim
echo -en "\nexecute pathogen#infect()\n" >> ~/.vimrc

git clone https://github.com/chr4/nginx.vim ~/.vim/bundle/nginx.vim

# 2) Set location of NGINX config files:
cat > ~/.vim/filetype.vim << __EOF__
au BufRead,BufNewFile /etc/nginx/*,/etc/nginx/conf.d/*,/usr/local/nginx/conf/*,*/conf/nginx.conf if &ft == '' | setfiletype nginx | endif
__EOF__
Sublime Text

Install cabal - system for building and packaging Haskell libraries and programs (on Ubuntu):

add-apt-repository -y ppa:hvr/ghc
apt-get update

apt-get install -y cabal-install-1.22 ghc-7.10.2

# Add this to your shell main configuration file:
export PATH=$HOME/.cabal/bin:/opt/cabal/1.22/bin:/opt/ghc/7.10.2/bin:$PATH
source $HOME/.<shellrc>

cabal update
  • nginx-lint:

    git clone https://github.com/temoto/nginx-lint
    
    cd nginx-lint && cabal install --global
  • sublime-nginx + SublimeLinter-contrib-nginx-lint:

    Bring up the Command Palette and type install. Among the commands you should see Package Control: Install Package. Type nginx to install sublime-nginx and after that do the above again for install SublimeLinter-contrib-nginx-lint: type SublimeLinter-contrib-nginx-lint.

Nginx processes

NGINX has one master process and one or more worker processes.

The main purposes of the master process is to read and evaluate configuration files, as well as maintain the worker processes (respawn when a worker dies), handle signals, notify workers, opens log files, and, of course binding to ports.

Master process should be started as root user, because this will allow NGINX to open sockets below 1024 (it needs to be able to listen on port 80 for HTTP and 443 for HTTPS).

The worker processes do the actual processing of requests and get commands from master process. They runs in an event loop, handle network connections, read and write content to disk, and communicate with upstream servers. These are spawned by the master process, and the user and group will as specified (unprivileged).

NGINX has also cache loader and cache manager processes but only if you enable caching.

The following signals can be sent to the NGINX master process:

SIGNAL DESCRIPTION
TERM, INT quick shutdown
QUIT graceful shutdown
KILL halts a stubborn process
HUP configuration reload, start new workers, gracefully shutdown the old worker processes
USR1 reopen the log files
USR2 upgrade executable on the fly
WINCH gracefully shutdown the worker processes

There’s no need to control the worker processes yourself. However, they support some signals too:

SIGNAL DESCRIPTION
TERM, INT quick shutdown
QUIT graceful shutdown
USR1 reopen the log files

Connection processing

NGINX supports a variety of connection processing methods which depends on the platform used. For more information please see connection processing methods explanation.

Okay, so how many simultaneous connections can be processed by NGINX?

worker_processes * worker_connections = max clients (in theory)

According to this: if you are only running 2 worker processes with 512 worker connections, you will be able to serve 1024 clients.

It is a bit confusing because the sum of worker_processes and worker_connections does not directly translate into the number of clients that can be served simultaneously. Each clients (e.g. browsers) opens a number of parallel connections to download various components that compose a web page, for example, images, scripts, and so on.

Request processing stages

You may feel lost now (me too...) so I let myself put this great preview:

request-flow

This infographic comes from Inside NGINX official library.

Server blocks logic

NGINX does have Server Blocks (like a Virtual Hosts is an Apache) that use the listen and server_name directives to bind to tcp sockets.

Before start reading this chapter you should know what regular expressions are and how they works. I recommend two great and short write-ups about regular expressions created by Jonny Fox:

Why? Regular expressions can be used in both the server_name and location directives, and sometimes you must have a great skill of reading them. I think you should create the most readable regular expressions that do not become spaghetti code - impossible to debug and maintain.

It's short example of server block context (two server blocks):

http {

  index index.html;
  root /var/www/example.com/default;

  server {

    listen 10.10.250.10:80;
    server_name www.example.com;

    access_log logs/example.access.log main;

    root /var/www/example.com/public;

    ...

  }

  server {

    listen 10.10.250.11:80;
    server_name "~^(api.)?example\.com api.de.example.com";

    access_log logs/example.access.log main;

    proxy_pass http://localhost:8080;

    ...

  }

}
Handle incoming connections

NGINX uses the following logic to determining which virtual server (server block) should be used:

  1. Match the address:port pair to the listen directive - that can be multiple server blocks with listen directives of the same specificity that can handle the request

    NGINX use the address:port combination for handle incoming connections. This pair is assigned to the listen directive.

    The listen directive can be set to:

    • an IP address/port combination (127.0.0.1:80;)
    • a lone IP address, if only address is given, the port 80 is used (127.0.0.1;) - becomes 127.0.0.1:80;
    • a lone port which will listen to every interface on that port (80; or *:80;) - becomes 0.0.0.0:80;
    • the path to a UNIX-domain socket (unix:/var/run/nginx.sock;)

    If the listen directive is not present then either *:80 is used (runs with the superuser privileges), or *:8000 otherwise.

    The next steps are as follows:

    • NGINX translates all incomplete listen directives by substituting missing values with their default values (see above)

    • NGINX attempts to collect a list of the server blocks that match the request most specifically based on the address:port

    • If any block that is functionally using 0.0.0.0, will not be selected if there are matching blocks that list a specific IP

    • If there is only one most specific match, that server block will be used to serve the request

    • If there are multiple server blocks with the same level of matching, NGINX then begins to evaluate the server_name directive of each server block

    Look at this short example:

    # From client:
    GET / HTTP/1.0
    Host: api.random.com
    
    server {
    
      # This block will be processed:
      listen 192.168.252.10;  # --> 192.168.252.10:80
    
      ...
    
    }
    
    server {
    
      listen 80;  # --> *:80 --> 0.0.0.0:80
      server_name api.random.com;
    
      ...
    
    }
  2. Match the Host header field against the server_name directive as a string (the exact names hash table)

  3. Match the Host header field against the server_name directive with a wildcard at the beginning of the string (the hash table with wildcard names starting with an asterisk)

    If one is found, that block will be used to serve the request. If multiple matches are found, the longest match will be used to serve the request.

  4. Match the Host header field against the server_name directive with a wildcard at the end of the string (the hash table with wildcard names ending with an asterisk)

    If one is found, that block is used to serve the request. If multiple matches are found, the longest match will be used to serve the request.

  5. Match the Host header field against the server_name directive as a regular expression

    The first server_name with a regular expression that matches the Host header will be used to serve the request.

  6. If all the Host headers doesn't match, then direct to the listen directive marked as default_server

  7. If all the Host headers doesn't match and there is no default_server, direct to the first server with a listen directive that satisfies first step

  8. Finally, NGINX goes to the location context

This short list is based on Mastering Nginx - The virtual server section.

Matching location

For each request, NGINX goes through a process to choose the best location block that will be used to serve that request.

Let's short introduction something about this:

  • the exact match is the best priority (processed first); ends search if match
  • the prefix match is the second priority; there are two types of prefixes: ^~ and (none), if this match used the ^~ prefix, searching stops
  • the regular expression match has the lowest priority; there are two types of prefixes: ~ and ~*; in the order they are defined in the configuration file
  • if regular expression searching yielded a match, that result is used, otherwise, the match from prefix searching is used

Short example from the Nginx documentation:

location = / {
  # Matches the query / only.
  [ configuration A ]
}
location / {
  # Matches any query, since all queries begin with /, but regular
  # expressions and any longer conventional blocks will be
  # matched first.
  [ configuration B ]
}
location /documents/ {
  # Matches any query beginning with /documents/ and continues searching,
  # so regular expressions will be checked. This will be matched only if
  # regular expressions don't find a match.
  [ configuration C ]
}
location ^~ /images/ {
  # Matches any query beginning with /images/ and halts searching,
  # so regular expressions will not be checked.
  [ configuration D ]
}
location ~* \.(gif|jpg|jpeg)$ {
  # Matches any request ending in gif, jpg, or jpeg. However, all
  # requests to the /images/ directory will be handled by
  # Configuration D.
  [ configuration E ]
}

To help you understand how does the NGINX location match works:

The location syntax looks like:

location optional_modifier location_match { ... }

location_match in the above defines what NGINX should check the request URI against. The optional_modifier below will cause the associated location block to be interpreted as follows:

  • (none): if no modifiers are present, the location is interpreted as a prefix match. To determine a match, the location will now be matched against the beginning of the URI

  • =: is an exact match, without any wildcards, prefix matching or regular expressions; forces a literal match between the request URI and the location parameter

  • ~: if a tilde modifier is present, this location must be used for case sensitive matching (RE match)

  • ~*: if a tilde and asterisk modifier is used, the location must be used for case insensitive matching (RE match)

  • ^~: assuming this block is the best non-RE match, a carat followed by a tilde modifier means that RE matching will not take place

The process of choosing NGINX location block is as follows:

  1. Prefix-based NGINX location matches (no regular expression). Each location will be checked against the request URI

  2. NGINX searches for an exact match. If a "=" modifier exactly matches the request URI, this specific location block is chosen right away

  3. If no exact (meaning no "=" modifier) location block is found, NGINX will continue with non-exact prefixes. It starts with the longest matching prefix location for this URI, with the following approach:

    • In case the longest matching prefix location has the "^~" modifier, NGINX will stop its search right away and choose this location

    • Assuming the longest matching prefix location doesn’t use the "^~"modifier, the match is temporarily stored and the process continues

  4. As soon as the longest matching prefix location is chosen and stored, NGINX continues to evaluate the case-sensitive and insensitive regular expression locations. The first regular expression location that fits the URI is selected right away to process the request

  5. If no regular expression locations are found that match the request URI, the previously stored prefix location is selected to serve the request

In order to better understand how this process work please see this short cheatsheet that will allow you to design your location blocks in a predictable way:

nginx-location-cheatsheet

I recommend to use external tools for testing regular expressions. For more please see online tools chapter.

Ok, so we have following more complicated configuration:

server {

 listen           80;
 server_name      xyz.com www.xyz.com;

 location ~ ^/(media|static)/ {
  root            /var/www/xyz.com/static;
  expires         10d;
 }

 location ~* ^/(media2|static2) {
  root            /var/www/xyz.com/static2;
  expires         20d;
 }

 location /static3 {
  root            /var/www/xyz.com/static3;
 }

 location ^~ /static4 {
  root            /var/www/xyz.com/static4;
 }

 location = /api {
  proxy_pass      http://127.0.0.1:8080;
 }

 location / {
  proxy_pass      http://127.0.0.1:8080;
 }

 location /backend {
  proxy_pass      http://127.0.0.1:8080;
 }

 location ~ logo.xcf$ {
  root            /var/www/logo;
  expires         48h;
 }

 location ~* .(png|ico|gif|xcf)$ {
  root            /var/www/img;
  expires         24h;
 }

 location ~ logo.ico$ {
  root            /var/www/logo;
  expires         96h;
 }

 location ~ logo.jpg$ {
  root            /var/www/logo;
  expires         48h;
 }

}

And here is the table with the results:

URL LOCATIONS FOUND FINAL MATCH
/ 1) prefix match for / /
/css 1) prefix match for / /
/api 1) exact match for /api /api
/api/ 1) prefix match for / /
/backend 1) prefix match for /
2) prefix match for /backend
/backend
/static 1) prefix match for / /
/static/header.png 1) prefix match for /
2) case sensitive regex match for ^/(media|static)/
^/(media|static)/
/static/logo.jpg 1) prefix match for /
2) case sensitive regex match for ^/(media|static)/
^/(media|static)/
/media2 1) prefix match for /
2) case insensitive regex match for ^/(media2|static2)
^/(media2|static2)
/media2/ 1) prefix match for /
2) case insensitive regex match for ^/(media2|static2)
^/(media2|static2)
/static2/logo.jpg 1) prefix match for /
2) case insensitive regex match for ^/(media2|static2)
^/(media2|static2)
/static2/logo.png 1) prefix match for /
2) case insensitive regex match for ^/(media2|static2)
^/(media2|static2)
/static3/logo.jpg 1) prefix match for /static3
2) prefix match for /
3) case sensitive regex match for logo.jpg$
logo.jpg$
/static3/logo.png 1) prefix match for /static3
2) prefix match for /
3) case insensitive regex match for .(png|ico|gif)$
.(png|ico|gif|xcf)$
/static4/logo.jpg 1) priority prefix match for /static4
2) prefix match for /
/static4
/static4/logo.png 1) priority prefix match for /static4
2) prefix match for /
/static4
/static5/logo.jpg 1) prefix match for /
2) case sensitive regex match for logo.jpg$
logo.jpg$
/static5/logo.png 1) prefix match for /
2) prefix match for /
.(png|ico|gif|xcf)$
/static5/logo.xcf 1) prefix match for /
2) case sensitive regex match for logo.xcf$
logo.xcf$
/static5/logo.ico 1) prefix match for /
2) prefix match for /
.(png|ico|gif|xcf)$

Error log severity levels

The following is a list of all severity levels:

TYPE DESCRIPTION
debug information that can be useful to pinpoint where a problem is occurring
info informational messages that aren’t necessary to read but may be good to know
notice something normal happened that is worth noting
warn something unexpected happened, however is not a cause for concern
error something was unsuccessful, contains the action of limiting rules
crit important problems that need to be addressed
alert severe situation where action is needed promptly
emerg the system is in an unusable state and requires immediate attention

For example: if you set crit error log level, messages of crit, alert, and emerg levels are logged.

Rate limiting

All rate limiting rules (definitions) should be added to the NGINX http context.

Rate limiting rules are useful for:

  • traffic shaping
  • traffic optimizing
  • slow down the rate of incoming requests
  • protect http requests flood
  • protect against slow http attacks
  • prevent consume a lot of bandwidth
  • mitigating ddos attacks
  • protect brute-force attacks

NGINX has following variables (unique keys) that can be used in a rate limiting rules:

VARIABLE DESCRIPTION
$remote_addr client address
$binary_remote_addr client address in a binary form, it is smaller and saves space then remote_addr
$server_name name of the server which accepted a request
$request_uri full original request URI (with arguments)
$query_string arguments in the request line

Please see official doc for more information about variables.

NGINX also provides following keys:

KEY DESCRIPTION
limit_req_zone stores the current number of excessive requests
limit_conn_zone stores the maximum allowed number of connections

and directives:

DIRECTIVE DESCRIPTION
limit_req sets the shared memory zone and the maximum burst size of requests
limit_conn sets the shared memory zone and the maximum allowed number of connections to the server per a client IP

Keys are used to store the state of each IP address and how often it has accessed a limited object. This information are stored in shared memory available from all NGINX worker processes.

Both keys also provides response status parameters indicating too many requests or connections with specific http code (default 503).

  • limit_req_status <value>
  • limit_conn_status <value>

For example, if you want to set the desired logging level for cases when the server limits the number of connections:

# Add this to http context:
limit_req_status 429;

# Set your own error page for 429 http code:
error_page 429 /rate_limit.html;
location = /rate_limit.html {
  root /usr/share/www/http-error-pages/sites/other;
  internal;
}

# And create this file:
cat > /usr/share/www/http-error-pages/sites/other/rate_limit.html << __EOF__
HTTP 429 Too Many Requests
__EOF__

Rate limiting rules also have zones that lets you define a shared space in which to count the incoming requests or connections.

All requests or connections coming into the same space will be counted in the same rate limit. This is what allows you to limit per URL, per IP, or anything else.

The zone has two required parts:

  • <name> - is the zone identifier
  • <size> - is the zone size

Example:

<key> <variable> zone=<name>:<size>;

State information for about 16,000 IP addresses takes 1 megabyte. So 1 kilobyte zone has 16 IP addresses.

The range of zones is as follows:

  • http context
http {

  ... zone=<name>;
  • server context
server {

  ... zone=<name>;
  • location directive
location /api {

  ... zone=<name>;

limit_req_zone key lets you set rate parameter (optional) - it defines the rate limited URL(s).

For enable queue you should use limit_req or limit_conn directives (see above). limit_req also provides optional parameters:

PARAMETER DESCRIPTION
burst=<num> sets the maximum number of excessive requests that await to be processed in a timely manner; maximum requests as rate * burst in burst seconds
nodelay it imposes a rate limit without constraining the allowed spacing between requests; default NGINX would return 503 response and not handle excessive requests

nodelay parameters are only useful when you also set a burst.

Without nodelay option NGINX would wait (no 503 response) and handle excessive requests with some delay.

Analyse configuration

It is an essential way for testing NGINX configuration:

nginx -t -c /etc/nginx/nginx.conf;

An external tool for analyse NGINX configuration is gixy:

gixy /etc/nginx/nginx.conf

Monitoring

GoAccess

Paths configuration file:

  • /etc/goaccess.conf
  • /etc/goaccess/goaccess.conf
  • /usr/local/etc/goaccess.conf

Prior to start GoAccess enable these parameters:

time-format %H:%M:%S
date-format %d/%b/%Y
log-format  %h %^[%d:%t %^] "%r" %s %b "%R" "%u"
Analyse log file and enable all recorded statistics
goaccess -f /path/to/logfile -a
Analyse compressed log file
zcat /path/to/logfile.1.gz | goaccess -a -p /etc/goaccess/goaccess.conf
Analyse log file remotely
ssh user@remote_host '/path/to/logfile' | goaccess -a
Analyse log file and generate html report
goaccess -p /etc/goaccess/goaccess.conf -f /path/to/logfile --log-format=COMBINED -o /var/www/index.html
Ngxtop
Analyse log file
ngxtop -l /path/to/logfile
Analyse log file and print requests with 4xx and 5xx
ngxtop -l /path/to/logfile -i 'status >= 400' print request status
Analyse log file remotely
ssh user@remote_host tail -f /path/to/logfile | ngxtop -f combined

Debugging

Check if the module has been compiled
nginx -V 2>&1 | grep -- 'http_geoip_module'
See the top 5 IP addresses in a web server log
cut -d ' ' -f1 /path/to/logfile | sort | uniq -c | sort -nr | head -5 | nl
Analyse web server log and show only 2xx http codes
tail -n 100 -f /path/to/logfile | grep "HTTP/[1-2].[0-1]\" [2]"
Analyse web server log and show only 5xx http codes
tail -n 100 -f /path/to/logfile | grep "HTTP/[1-2].[0-1]\" [5]"
Calculating requests per second
# In real time:
tail -F /path/to/logfile | pv -lr >/dev/null

awk '{print $4}' /path/to/logfile | uniq -c | sort -rn | head | tr -d "["
Get range of dates in a web server log
# 1)
awk '/05\/Feb\/2019:09:2.*/,/05\/Feb\/2019:09:5.*/' /path/to/logfile

# 2)
awk '/'$(date -d "1 hours ago" "+%d\\/%b\\/%Y:%H:%M")'/,/'$(date "+%d\\/%b\\/%Y:%H:%M")'/ { print $0 }' /path/to/logfile
Get line rates from web server log
tail -F /path/to/logfile | pv -N RAW -lc 1>/dev/null
Trace network traffic for all Nginx processes
strace -e trace=network -p `pidof nginx | sed -e 's/ /,/g'`
List all files accessed by a Nginx
strace -ff -e trace=file nginx 2>&1 | perl -ne 's/^[^"]+"(([^\\"]|\\[\\"nt])*)".*/$1/ && print'
Check that the gzip_static module is working
strace -p `pidof nginx | sed -e 's/ /,/g'` 2>&1 | grep gz
Which worker processing current request

Example 1 (more elegant way):

log_format debug-req-trace
                '$pid - "$request_method $scheme://$host$request_uri" '
                '$remote_addr:$remote_port $server_addr:$server_port '
                '$request_id';

# Output example:
31863 - "GET https://example.com/" 35.228.233.xxx:63784 10.240.20.2:443 be90154db5beb0e9dd13c5d91c8ecd4c

Example 2:

# Run strace in the background:
nohup strace -s 256 -p `pidof nginx | sed -e 's/ /,/g'` 2>&1 -o /tmp/nginx-req.trace </dev/null >/dev/null 2>/dev/null &

# Watch output file:
watch -n 0.1 "awk '/Host:/ {print \"pid: \" \$1 \", \" \"host: \" \$6}' /tmp/nginx-req.trace | sed 's/\\\r\\\n.*//'"

# Output example:
Every 0.1s: awk '/Host:/ {print "pid: " $1 ", " "host: " $6}' /tmp/nginx-req.trace | sed 's/\\r\\n.*//'

pid: 31863, host: example.com

Shell aliases

alias ng.test='nginx -t -c /etc/nginx/nginx.conf'

alias ng.stop='ng.test && systemctl stop nginx'

alias ng.reload='ng.test && systemctl reload nginx'
alias ng.reload='ng.test && kill -HUP $(cat /var/run/nginx.pid)'
#                       ... kill -HUP $(ps auxw | grep [n]ginx | grep master | awk '{print $2}')

alias ng.restart='ng.test && systemctl restart nginx'
alias ng.restart='ng.test && kill -QUIT $(cat /var/run/nginx.pid) && /usr/sbin/nginx'
#                        ... kill -QUIT $(ps auxw | grep [n]ginx | grep master | awk '{print $2}') ...

Configuration snippets

Restricting access with basic authentication
# 1) Generate file with htpasswd command:
htpasswd -c htpasswd_example.com.conf <username>

# 2) Include this file in specific context: (e.g. server):
server_name example.com;

  ...

  # These directives are optional, only if we need them:
  satisfy all;

  deny    10.255.10.0/24;
  allow   192.168.0.0/16;
  allow   127.0.0.1;
  deny    all;

  # It's important:
  auth_basic            "Restricted Area";
  auth_basic_user_file  /etc/nginx/acls/htpasswd_example.com.conf;

  location / {

    ...

  location /public/ {

    auth_basic off;

  }

  ...
Blocking/allowing IP addresses

Example 1:

# 1) File: /etc/nginx/acls/allow.map.conf

# Map module:
map $remote_addr $globals_internal_map_acl {

  # Status code:
  #  - 0 = false
  #  - 1 = true
  default 0;

  ### INTERNAL ###
  10.255.10.0/24 1;
  10.255.20.0/24 1;
  10.255.30.0/24 1;
  192.168.0.0/16 1;

}

# 2) Include this file in http context:
include /etc/nginx/acls/allow.map.conf;

# 3) Turn on in a specific context (e.g. location):
server_name example.com;

  ...

  location / {

    proxy_pass http://localhost:80;
    client_max_body_size 10m;

  }

  location ~ ^/(backend|api|admin) {

    if ($globals_internal_map_acl) {

      set $pass 1;

    }

    if ($pass = 1) {

      proxy_pass http://localhost:80;
      client_max_body_size 10m;

    }

    if ($pass != 1) {

      rewrite ^(.*) https://example.com;

    }

  ...

Example 2:

# 1) File: /etc/nginx/acls/allow.geo.conf

# Geo module:
geo $globals_internal_geo_acl {

  # Status code:
  #  - 0 = false
  #  - 1 = true
  default 0;

  ### INTERNAL ###
  10.255.10.0/24 1;
  10.255.20.0/24 1;
  10.255.30.0/24 1;
  192.168.0.0/16 1;

}

# 2) Include this file in http context:
include /etc/nginx/acls/allow.geo.conf;

# 3) Turn on in a specific context (e.g. location):
server_name example.com;

  ...

  location / {

    proxy_pass http://localhost:80;
    client_max_body_size 10m;

  }

  location ~ ^/(backend|api|admin) {

    if ($globals_internal_geo_acl = 0) {

      return 403;

    }

    proxy_pass http://localhost:80;
    client_max_body_size 10m;

  ...

Example 3:

# 1) File: /etc/nginx/acls/allow.conf

### INTERNAL ###
allow 10.255.10.0/24;
allow 10.255.20.0/24;
allow 10.255.30.0/24;
allow 192.168.0.0/16;

### EXTERNAL ###
allow 35.228.233.xxx;

# 2) Include this file in http context:
include /etc/nginx/acls/allow.conf;

# 3) Turn on in a specific context (e.g. server):
server_name example.com;

  include /etc/nginx/acls/allow.conf;
  allow   35.228.233.xxx;
  deny    all;

  ...
Blocking referrer spam

Example 1:

# 1) File: /etc/nginx/limits.conf
map $http_referer $invalid_referer {

  hostnames;

  default                   0;

  # Invalid referrers:
  "invalid.com"             1;
  "~*spamdomain4.com"       1;
  "~*.invalid\.org"         1;

}

# 2) Include this file in http context:
include /etc/nginx/limits.conf;

# 3) Turn on in a specific context (e.g. server):
server_name example.com;

  if ($invalid_referer) { return 403; }

  ...

Example 2:

# 1) Turn on in a specific context (e.g. location):
location /check_status {

  if ($http_referer ~ "spam1\.com|spam2\.com|spam3\.com") {

    return 444;

  }

  ...

How to test?

siege -b -r 2 -c 40 -v https:/example.com/storage/img/header.jpg -H "Referer: https://spamdomain4.com/"
** SIEGE 4.0.4
** Preparing 5 concurrent users for battle.
The server is now under siege...
HTTP/1.1 403     0.11 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.12 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.18 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.18 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.19 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.10 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.11 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.11 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.12 secs:     124 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 403     0.12 secs:     124 bytes ==> GET  /storage/img/header.jpg

...
Limiting referrer spam

Example 1:

# 1) File: /etc/nginx/limits.conf
map $http_referer $limit_ip_key_by_referer {

  hostnames;

  # It's important because if you set numeric value, e.g. 0 rate limiting rule will be catch all referers:
  default                   "";

  # Invalid referrers (we restrict them):
  "invalid.com"             $binary_remote_addr;
  "~referer-xyz.com"        $binary_remote_addr;
  "~*spamdomain4.com"       $binary_remote_addr;
  "~*.invalid\.org"         $binary_remote_addr;

}

limit_req_zone $limit_ip_key_by_referer zone=req_for_remote_addr_by_referer:1m rate=5r/s;

# 2) Include this file in http context:
include /etc/nginx/limits.conf;

# 3) Turn on in a specific context (e.g. server):
server_name example.com;

  limit_req zone=req_for_remote_addr_by_referer burst=2;

  ...

How to test?

siege -b -r 2 -c 40 -v https:/example.com/storage/img/header.jpg -H "Referer: https://spamdomain4.com/"
** SIEGE 4.0.4
** Preparing 5 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200     0.13 secs:    3174 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 503     0.14 secs:     206 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 503     0.15 secs:     206 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 503     0.10 secs:     206 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 503     0.10 secs:     206 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 503     0.10 secs:     206 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 200     0.63 secs:    3174 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 200     1.13 secs:    3174 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 200     1.00 secs:    3174 bytes ==> GET  /storage/img/header.jpg
HTTP/1.1 200     1.04 secs:    3174 bytes ==> GET  /storage/img/header.jpg

...
Limiting the rate of requests with burst mode
limit_req_zone $binary_remote_addr zone=req_for_remote_addr:64k rate=10r/m;
  • key/zone type: limit_req_zone
  • the unique key for limiter: $binary_remote_addr
    • limit requests per IP as following
  • zone name: req_for_remote_addr
  • zone size: 64k (1024 IP addresses)
  • rate is 0,16 request each second or 10 requests per minute (1 request every 6 second)

Example of use:

location ~ /stats {

  limit_req zone=req_for_remote_addr burst=5;
  • set maximum requests as rate * burst in burst seconds
    • with bursts not exceeding 5 requests:
      • 0,16r/s * 5 = 0.80 requests per 5 seconds
      • 10r/m * 5 = 50 requests per 5 minutes

Testing queue:

# siege -b -r 1 -c 12 -v https://x409.info/stats/
** SIEGE 4.0.4
** Preparing 12 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200 *   0.20 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 503     0.20 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.20 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.21 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.22 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.22 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.23 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 200 *   6.22 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *  12.24 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *  18.27 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *  24.30 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *  30.32 secs:       2 bytes ==> GET  /stats/
             |
             - burst=5
             - 0,16r/s, 10r/m - 1r every 6 seconds

Transactions:              6 hits
Availability:          50.00 %
Elapsed time:          30.32 secs
Data transferred:       0.01 MB
Response time:         15.47 secs
Transaction rate:       0.20 trans/sec
Throughput:             0.00 MB/sec
Concurrency:            3.06
Successful transactions:   6
Failed transactions:       6
Longest transaction:   30.32
Shortest transaction:   0.20
Limiting the rate of requests with burst mode and nodelay
limit_req_zone $binary_remote_addr zone=req_for_remote_addr:50m rate=2r/s;
  • key/zone type: limit_req_zone
  • the unique key for limiter: $binary_remote_addr
    • limit requests per IP as following
  • zone name: req_for_remote_addr
  • zone size: 50m (800,000 IP addresses)
  • rate is 2 request each second or 120 requests per minute (2 requests every 1 second)

Example of use:

location ~ /stats {

  limit_req zone=req_for_remote_addr burst=5 nodelay;
  • set maximum requests as rate * burst in burst seconds
    • with bursts not exceeding 5 requests
      • 2r/s * 5 = 10 requests per 5 seconds
      • 120r/m * 5 = 600 requests per 5 minutes
  • allocates slots in the queue according to the burst parameter with nodelay

Testing queue:

# siege -b -r 1 -c 12 -v https://x409.info/stats/
** SIEGE 4.0.4
** Preparing 12 concurrent users for battle.
The server is now under siege...
HTTP/1.1 200 *   0.18 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *   0.18 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *   0.19 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *   0.19 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *   0.19 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 200 *   0.19 secs:       2 bytes ==> GET  /stats/
HTTP/1.1 503     0.19 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.19 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.20 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.21 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.21 secs:    1501 bytes ==> GET  /stats/
HTTP/1.1 503     0.22 secs:    1501 bytes ==> GET  /stats/
             |
             - burst=5 with nodelay
             - 2r/s, 120r/m - 1r every 0.5 second

Transactions:              6 hits
Availability:          50.00 %
Elapsed time:           0.23 secs
Data transferred:       0.01 MB
Response time:          0.39 secs
Transaction rate:      26.09 trans/sec
Throughput:             0.04 MB/sec
Concurrency:           10.17
Successful transactions:   6
Failed transactions:       6
Longest transaction:    0.22
Shortest transaction:   0.18
Limiting the number of connections
limit_conn_zone $binary_remote_addr zone=conn_for_remote_addr:1m;
  • key/zone type: limit_conn_zone
  • the unique key for limiter: $binary_remote_addr
    • limit requests per IP as following
  • zone name: conn_for_remote_addr
  • zone size: 1m (16,000 IP addresses)

Example of use:

location ~ /stats {

  limit_conn conn_for_remote_addr 1;
  • limit a single IP address to make no more than 1 connection from IP at the same time

Testing queue:

# siege -b -r 1 -c 100 -t 10s --no-parser https://x409.info/stats/
defaulting to time-based testing: 10 seconds
** SIEGE 4.0.4
** Preparing 100 concurrent users for battle.
The server is now under siege...
Lifting the server siege...
Transactions:            364 hits
Availability:          32.13 %
Elapsed time:           9.00 secs
Data transferred:       1.10 MB
Response time:          2.37 secs
Transaction rate:      40.44 trans/sec
Throughput:             0.12 MB/sec
Concurrency:           95.67
Successful transactions: 364
Failed transactions:     769
Longest transaction:    1.10
Shortest transaction:   0.38
Adding and removing the www prefix
  • www to non-www:
server {

  ...

  server_name www.domain.com;

  # $scheme will get the http or https protocol:
  return 301 $scheme://domain.com$request_uri;

}
  • non-www to www:
server {

  ...

  server_name domain.com;

  # $scheme will get the http or https protocol:
  return 301 $scheme://www.domain.com$request_uri;

}
Rewrite POST request with payload to external endpoint

POST data is passed in the body of the request, which gets dropped if you do a standard redirect.

Look at this:

DESCRITPION PERMANENT TEMPORARY
allows changing the request method from POST to GET 301 302
does not allow changing the request method from POST to GET 308 307

You can try with the HTTP status code 307, a RFC compliant browser should repeat the post request. You just need to write a NGINX rewrite rule with HTTP status code 307 or 308:

location /api {

  # HTTP 307 only for POST requests:
  if ($request_method = POST) {

    return 307 https://api.example.com?request_uri;

  }

  # You can keep this for non-POST requests:
  rewrite ^ https://api.example.com?request_uri permanent;

  client_max_body_size    10m;

  ...

}
Allow multiple cross-domains using the CORS headers

Example 1:

location ~* \.(?:ttf|ttc|otf|eot|woff|woff2)$ {

  if ( $http_origin ~* (https?://(.+\.)?(domain1|domain2|domain3)\.(?:me|co|com)$) ) {

    add_header "Access-Control-Allow-Origin" "$http_origin";

  }

}

Example 2 (more slightly configuration; for GETs and POSTs):

location / {

  if ($http_origin ~* (^https?://([^/]+\.)*(domainone|domaintwo)\.com$)) {

    set $cors "true";

  }

  # Determine the HTTP request method used:
  if ($request_method = 'GET') {

    set $cors "${cors}get";

  }

  if ($request_method = 'POST') {

    set $cors "${cors}post";

  }

  if ($cors = "true") {

    # Catch all in case there's a request method we're not dealing with properly:
    add_header 'Access-Control-Allow-Origin' "$http_origin";

  }

  if ($cors = "trueget") {

    add_header 'Access-Control-Allow-Origin' "$http_origin";
    add_header 'Access-Control-Allow-Credentials' 'true';
    add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';

  }

  if ($cors = "truepost") {

    add_header 'Access-Control-Allow-Origin' "$http_origin";
    add_header 'Access-Control-Allow-Credentials' 'true';
    add_header 'Access-Control-Allow-Methods' 'GET, POST, OPTIONS';
    add_header 'Access-Control-Allow-Headers' 'DNT,X-CustomHeader,Keep-Alive,User-Agent,X-Requested-With,If-Modified-Since,Cache-Control,Content-Type';

  }

}

Installation from prebuilt packages

RHEL7 or CentOS 7
From EPEL
# Install epel repository:
yum install epel-release
# or alternative:
#   wget -c https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
#   yum install epel-release-latest-7.noarch.rpm

# Install NGINX:
yum install nginx
From Software Collections
# Install and enable scl:
yum install centos-release-scl
yum-config-manager --enable rhel-server-rhscl-7-rpms

# Install NGINX (rh-nginx14, rh-nginx16, rh-nginx18):
yum install rh-nginx16

# Enable NGINX from SCL:
scl enable rh-nginx16 bash
From Official Repository
# Where:
#   - <os_type> is: rhel or centos
cat > /etc/yum.repos.d/nginx.repo << __EOF__
[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/<os_type>/$releasever/$basearch/
gpgcheck=0
enabled=1
__EOF__

# Install NGINX:
yum install nginx
Debian or Ubuntu

Check available flavors of NGINX before install. For more information please see this great answer by Thomas Ward.

From Debian/Ubuntu Repository
# Install NGINX:
apt-get install nginx
From Official Repository
# Where:
#   - <os_type> is: debian or ubuntu
#   - <os_release> is: xenial, bionic, jessie, stretch or other
cat > /etc/apt/sources.list.d/nginx.list << __EOF__
deb http://nginx.org/packages/<os_type>/ <os_release> nginx
deb-src http://nginx.org/packages/<os_type>/ <os_release> nginx
__EOF__

# Update packages list:
apt-get update

# Download the public key (or <pub_key> from your GPG error):
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys <pub_key>

# Install NGINX:
apt-get update
apt-get install nginx

Installation from source

Before the beginning installation process please see two very important articles:

Look also on this short note about the system locations. That can be useful too:

  • For booting the system, rescues and maintenance: /
    • /bin - user programs
    • /sbin - system programs
    • /lib - shared libraries
  • Full running environment: /usr
    • /usr/bin - user programs
    • /usr/sbin - system programs
    • /usr/lib - shared libraries
    • /usr/share - manual pages, data
  • Added packages: /usr/local
    • /usr/local/bin - user programs
    • /usr/local/sbin - system programs
    • /usr/local/lib - shared libraries
    • /usr/local/share - manual pages, data
Nginx package

There are currently two versions of NGINX:

  • stable - is recommended, doesn’t include all of the latest features, but has critical bug fixes from mainline release
  • mainline - is typically quite stable as well, includes the latest features and bug fixes and is always up to date

You can download NGINX source code from an official read-only mirrors:

Detailed instructions about download and compile the NGINX sources can be found later in the handbook.

Dependencies

Mandatory requirements:

Download, compile and install or install prebuilt packages from repository of your distribution.

OpenResty's LuaJIT uses its own branch of LuaJIT with various important bug fixes and optimizations for OpenResty's use cases.

I also use Cloudflare Zlib version due to performance. See below articles:

If you download and compile above sources the good point is to install additional packages (dependent on the system version) before building NGINX:

Debian Like RedHat Like Comment
gcc make build-essential bison gcc gcc-c++ kernel-devel bison
perl libperl-dev perl perl-ExtUtils-Embed
libssl-dev* openssl-devel*
zlib1g-dev* zlib-devel*
libpcre2-dev* pcre-devel*
libluajit-5.1-dev* luajit-devel*
libxslt-dev libxslt libxslt-devel
libgd-dev gd gd-devel
libgeoip-dev GeoIP-devel
libxml2-dev libxml2-dev
libexpat-dev expat-devel
libgoogle-perftools-dev
libgoogle-perftools4
gperftools-devel
cpio
gettext-devel
autoconf autoconf for jemalloc from sources
libjemalloc1
libjemalloc-dev*
jemalloc
jemalloc-devel*
for jemalloc
libpam0g-dev pam-devel for ngx_http_auth_pam_module

* If you don't use from sources.

Shell one-liners example:

# Ubuntu/Debian
apt-get install gcc make build-essential bison perl libperl-dev libssl-dev zlib1g-dev libpcre2-dev libluajit-5.1-dev libxslt-dev libgd-dev libgeoip-dev libxml2-dev libexpat-dev libgoogle-perftools-dev libgoogle-perftools4 autoconf

# RedHat/CentOS
yum install gcc gcc-c++ kernel-devel bison perl perl-ExtUtils-Embed openssl-devel zlib-devel pcre-devel luajit-devel libxslt libxslt-devel gd gd-devel GeoIP-devel libxml2-dev expat-devel gperftools-devel cpio gettext-devel autoconf
3rd party modules

Not all external modules can work properly with your currently NGINX version. You should read the documentation of each module before adding it to the modules list. You should also to check what version of module is compatible with your NGINX release.

Modules can be compiled as a shared object (*.so file) and then dynamically loaded into NGINX at runtime (--add-dynamic-module). On the other hand you can also built them into NGINX at compile time and linked to the NGINX binary statically (--add-module).

I mixed both variants because some of the modules are built-in automatically even if I try them to be compiled as a dynamic modules.

You can download external modules from:

A short description of the modules that I used (not only) in this step-by-step tutorial:

* Available in Tengine Web Server (but these modules may have been updated/patched by Tengine Team).
** Is already being used in quite a few third party modules.

Installation Nginx on Ubuntu 18.04

Pre installation tasks

Set the NGINX version (I use stable release):

export ngx_version="1.16.0"

Create directories:

mkdir /usr/local/src/nginx-${ngx_version}
mkdir /usr/local/src/nginx-${ngx_version}/master
mkdir /usr/local/src/nginx-${ngx_version}/modules
Install or build dependencies

In my configuration I used all prebuilt dependencies without libssl-dev, zlib1g-dev, libluajit-5.1-dev and libpcre2-dev because I compiled them manually - for TLS 1.3 support and with OpenResty recommendation for LuaJIT.

Install prebuilt packages, export variables and set symbolic link:

# It's important and required, regardless of chosen sources.
apt-get install gcc make build-essential bison perl libperl-dev libxslt-dev libgd-dev libgeoip-dev libxml2-dev libexpat-dev libgoogle-perftools-dev libgoogle-perftools4 autoconf

# In this example we use sources for all below packages so we do not install them:
apt-get install libssl-dev zlib1g-dev libpcre2-dev libluajit-5.1-dev

# For LuaJIT (libluajit-5.1-dev):
export LUA_LIB=/usr/local/x86_64-linux-gnu/
export LUA_INC=/usr/include/luajit-2.1/

ln -s /usr/lib/x86_64-linux-gnu/libluajit-5.1.so.2 /usr/local/lib/liblua.so

Remember to build sregex also if you use above steps.

Or download and compile them:

PCRE:

cd /usr/local/src/

wget https://ftp.pcre.org/pub/pcre/pcre-8.42.tar.gz && tar xzvf pcre-8.42.tar.gz

cd /usr/local/src/pcre-8.42

./configure

make -j2 && make test
make install

export PCRE_LIB=/usr/local/lib
export PCRE_INC=/usr/local/include
export PCRE_DIRECTORY="/usr/local/src/pcre-8.42"

Zlib:

# I recommend to use Cloudflare Zlib version (cloudflare/zlib) instead an original Zlib (zlib.net), but both installation methods are similar:
cd /usr/local/src/

# For original Zlib:
#   wget http://www.zlib.net/zlib-1.2.11.tar.gz && tar xzvf zlib-1.2.11.tar.gz
#   cd /usr/local/src/zlib-1.2.11

# For Cloudflare Zlib:
git clone --depth 1 https://github.com/cloudflare/zlib

cd /usr/local/src/zlib

./configure

make -j2 && make test
make install

export ZLIB_LIB=/usr/local/lib
export ZLIB_INC=/usr/local/include
export ZLIB_DIRECTORY="/usr/local/src/zlib"

OpenSSL:

cd /usr/local/src/

wget https://www.openssl.org/source/openssl-1.1.1b.tar.gz && tar xzvf openssl-1.1.1b.tar.gz

cd /usr/local/src/openssl-1.1.1b

./config --prefix=/usr/local/openssl-1.1.1b --openssldir=/usr/local/openssl-1.1.1b shared zlib no-ssl3 no-weak-ssl-ciphers

make -j2 && make test
make install

export OPENSSL_LIB=/usr/local/openssl-1.1.1b/lib
export OPENSSL_INC=/usr/local/openssl-1.1.1b/include
export OPENSSL_DIRECTORY="/usr/local/src/openssl-1.1.1b"

# Setup PATH environment variables:
cat > /etc/profile.d/openssl.sh << __EOF__
#!/bin/sh
export PATH=/usr/local/openssl-1.1.1b/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/openssl-1.1.1b/lib:${LD_LIBRARY_PATH}
__EOF__

chmod +x /etc/profile.d/openssl.sh && source /etc/profile.d/openssl.sh

# To make the OpenSSL 1.1.1b version visible globally first:
mv /usr/bin/openssl /usr/bin/openssl-1.1.0g
ln -s /usr/local/openssl-1.1.1b/bin/openssl /usr/bin/openssl

cat > /etc/ld.so.conf.d/openssl.conf << __EOF__
/usr/local/openssl-1.1.1b/lib
__EOF__

LuaJIT:

# I recommend to use OpenResty's branch (openresty/luajit2) instead LuaJIT (LuaJIT/LuaJIT), but both installation methods are similar:
cd /usr/local/src/

# For originall LuaJIT:
#   git clone http://luajit.org/git/luajit-2.0
#   cd /usr/local/src/luajit-2.0

# For OpenResty's LuaJIT:
git clone --depth 1 https://github.com/openresty/luajit2

cd /usr/local/src/luajit2

make && make install

export LUA_LIB=/usr/local/lib/
export LUA_INC=/usr/local/include/luajit-2.1/

ln -s /usr/local/lib/libluajit-5.1.so.2.1.0 /usr/local/lib/liblua.so

sregex:

Required for replace-filter-nginx-module module.

cd /usr/local/src/

git clone --depth 1 https://github.com/openresty/sregex

cd /usr/local/src/sregex

make && make install

jemalloc:

To verify jemalloc in use: lsof -n | grep jemalloc.

cd /usr/local/src/

git clone --depth 1 https://github.com/jemalloc/jemalloc

cd /usr/local/src/jemalloc

./autogen.sh

make && make install

export JEMALLOC_DIRECTORY="/usr/local/src/jemalloc"

Update links and cache to the shared libraries for both types of installation:

ldconfig
Get Nginx sources
cd /usr/local/src/nginx-${ngx_version}

wget https://nginx.org/download/nginx-${ngx_version}.tar.gz

# or alternative:
#   git clone --depth 1 https://github.com/nginx/nginx master

tar zxvf nginx-${ngx_version}.tar.gz -C /usr/local/src/nginx-${ngx_version}/master --strip 1
Download 3rd party modules
cd /usr/local/src/nginx-${ngx_version}/modules/

for i in \
https://github.com/simplresty/ngx_devel_kit \
https://github.com/openresty/lua-nginx-module \
https://github.com/openresty/set-misc-nginx-module \
https://github.com/openresty/echo-nginx-module \
https://github.com/openresty/headers-more-nginx-module \
https://github.com/openresty/replace-filter-nginx-module \
https://github.com/openresty/array-var-nginx-module \
https://github.com/openresty/encrypted-session-nginx-module \
https://github.com/vozlt/nginx-module-sysguard \
https://github.com/nginx-clojure/nginx-access-plus \
https://github.com/yaoweibin/ngx_http_substitutions_filter_module \
https://bitbucket.org/nginx-goodies/nginx-sticky-module-ng \
https://github.com/vozlt/nginx-module-vts \
https://github.com/google/ngx_brotli ; do

  git clone --depth 1 "$i"

done

wget http://mdounin.ru/hg/ngx_http_delay_module/archive/tip.tar.gz -O delay-module.tar.gz
mkdir delay-module && tar xzvf delay-module.tar.gz -C delay-module --strip 1

I also use some modules from Tengine:

  • ngx_backtrace_module
  • ngx_debug_pool
  • ngx_debug_timer
  • ngx_http_upstream_check_module
  • ngx_http_upstream_session_sticky_module
  • ngx_http_footer_filter_module
cd /usr/local/src/nginx-${ngx_version}/modules/

git clone --depth 1 https://github.com/alibaba/tengine

If you use NAXSI:

cd /usr/local/src/nginx-${ngx_version}/modules/

git clone --depth 1 https://github.com/nbs-system/naxsi
Build Nginx
cd /usr/local/src/nginx-${ngx_version}/master

# - you can also build NGINX without 3rd party modules
# - don't set values for --with-openssl, --with-pcre, and --with-zlib if you select prebuilt packages for them
./configure --prefix=/etc/nginx \
            --conf-path=/etc/nginx/nginx.conf \
            --sbin-path=/usr/sbin/nginx \
            --pid-path=/var/run/nginx.pid \
            --lock-path=/var/run/nginx.lock \
            --user=nginx \
            --group=nginx \
            --modules-path=/etc/nginx/modules \
            --error-log-path=/var/log/nginx/error.log \
            --http-log-path=/var/log/nginx/access.log \
            --http-client-body-temp-path=/var/cache/nginx/client_temp \
            --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
            --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
            --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
            --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
            --with-compat \
            --with-debug \
            --with-file-aio \
            --with-threads \
            --with-stream \
            --with-stream_realip_module \
            --with-stream_ssl_module \
            --with-stream_ssl_preread_module \
            --with-http_addition_module \
            --with-http_auth_request_module \
            --with-http_degradation_module \
            --with-http_geoip_module \
            --with-http_gunzip_module \
            --with-http_gzip_static_module \
            --with-http_image_filter_module \
            --with-http_perl_module \
            --with-http_random_index_module \
            --with-http_realip_module \
            --with-http_secure_link_module \
            --with-http_ssl_module \
            --with-http_stub_status_module \
            --with-http_sub_module \
            --with-http_v2_module \
            --with-google_perftools_module \
            --with-openssl=${OPENSSL_DIRECTORY} \
            --with-openssl-opt=no-weak-ssl-ciphers \
            --with-openssl-opt=no-ssl3 \
            --with-pcre=${PCRE_DIRECTORY} \
            --with-pcre-jit \
            --with-zlib=${ZLIB_DIRECTORY} \
            --without-http-cache \
            --without-http_memcached_module \
            --without-mail_pop3_module \
            --without-mail_imap_module \
            --without-mail_smtp_module \
            --without-http_fastcgi_module \
            --without-http_scgi_module \
            --without-http_uwsgi_module \
            --with-cc-opt='-I/usr/local/include -I/usr/local/openssl-1.1.1b/include -I/usr/local/include/luajit-2.1/ -I/usr/local/include/jemalloc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' \
            --with-ld-opt='-Wl,-E -L/usr/local/lib -ljemalloc -lpcre -Wl,-rpath,/usr/local/lib/,-z,relro -Wl,-z,now -pie' \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/ngx_devel_kit \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/encrypted-session-nginx-module \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/nginx-access-plus/src/c \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/ngx_http_substitutions_filter_module \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/nginx-sticky-module-ng \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/nginx-module-vts \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/ngx_brotli \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/tengine/modules/ngx_backtrace_module \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/tengine/modules/ngx_debug_pool \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/tengine/modules/ngx_debug_timer \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/tengine/modules/ngx_http_upstream_check_module \
            --add-module=/usr/local/src/nginx-${ngx_version}/modules/tengine/modules/ngx_http_footer_filter_module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/lua-nginx-module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/set-misc-nginx-module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/echo-nginx-module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/headers-more-nginx-module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/replace-filter-nginx-module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/array-var-nginx-module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/nginx-module-sysguard \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/delay-module \
            --add-dynamic-module=/usr/local/src/nginx-${ngx_version}/modules/naxsi/naxsi_src

make -j2 && make test
make install

ldconfig

Check NGINX version:

nginx -v
nginx version: nginx/1.16.0

And list all files in /etc/nginx:

.
├── fastcgi.conf
├── fastcgi.conf.default
├── fastcgi_params
├── fastcgi_params.default
├── html
│   ├── 50x.html
│   └── index.html
├── koi-utf
├── koi-win
├── mime.types
├── mime.types.default
├── modules
│   ├── ngx_http_array_var_module.so
│   ├── ngx_http_delay_module.so
│   ├── ngx_http_echo_module.so
│   ├── ngx_http_headers_more_filter_module.so
│   ├── ngx_http_lua_module.so
│   ├── ngx_http_replace_filter_module.so
│   ├── ngx_http_set_misc_module.so
│   └── ngx_http_sysguard_module.so
├── nginx.conf
├── nginx.conf.default
├── scgi_params
├── scgi_params.default
├── uwsgi_params
├── uwsgi_params.default
└── win-utf

2 directories, 25 files
Post installation tasks

Create a system user/group:

adduser --system --home /non-existent --no-create-home --shell /usr/sbin/nologin --disabled-login --disabled-password --gecos "nginx user" --group nginx

Create required directories:

for i in \
/var/www \
/var/log/nginx \
/var/cache/nginx ; do

  mkdir -p "$i" && chown -R nginx:nginx "$i"

done

Add systemd service:

cat > /lib/systemd/system/nginx.service << __EOF__
# Stop dance for nginx
# =======================
#
# ExecStop sends SIGSTOP (graceful stop) to the nginx process.
# If, after 5s (--retry QUIT/5) nginx is still running, systemd takes control
# and sends SIGTERM (fast shutdown) to the main process.
# After another 5s (TimeoutStopSec=5), and if nginx is alive, systemd sends
# SIGKILL to all the remaining processes in the process group (KillMode=mixed).
#
# nginx signals reference doc:
# http://nginx.org/en/docs/control.html
#
[Unit]
Description=A high performance web server and a reverse proxy server
Documentation=man:nginx(8)
After=network.target

[Service]
Type=forking
PIDFile=/run/nginx.pid
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
ExecReload=/usr/sbin/nginx -g 'daemon on; master_process on;' -s reload
ExecStop=-/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid
TimeoutStopSec=5
KillMode=mixed

[Install]
WantedBy=multi-user.target
__EOF__

Reload systemd manager configuration:

systemctl daemon-reload

Enable NGINX service:

systemctl enable nginx

Test NGINX configuration:

nginx -t -c /etc/nginx/nginx.conf

Installation Tengine on Ubuntu 18.04

Tengine is a web server originated by Taobao, the largest e-commerce website in Asia. It is based on the NGINX HTTP server and has many advanced features. There’s a lot of features in Tengine that do not (yet) exist in NGINX.

Generally, Tengine is a great solution, including many patches, improvements, additional modules, and most importantly it is very actively maintained.

The build and installation process is very similar to Install Nginx on Ubuntu 18.04. However, I will only specify the most important changes.

Pre installation tasks

Create directories:

mkdir /usr/local/src/tengine
mkdir /usr/local/src/tengine/master
mkdir /usr/local/src/tengine/modules
Install or build dependencies

Install prebuilt packages, export variables and set symbolic link:

apt-get install gcc make build-essential bison perl libperl-dev libxslt-dev libgd-dev libgeoip-dev libxml2-dev libexpat-dev libgoogle-perftools-dev libgoogle-perftools4 autoconf

# In this example we don't use zlib sources:
apt-get install zlib1g-dev

PCRE:

cd /usr/local/src/

wget https://ftp.pcre.org/pub/pcre/pcre-8.42.tar.gz && tar xzvf pcre-8.42.tar.gz

cd /usr/local/src/pcre-8.42

./configure

make -j2 && make test
make install

export PCRE_LIB=/usr/local/lib
export PCRE_INC=/usr/local/include
export PCRE_DIRECTORY="/usr/local/src/pcre-8.42"

OpenSSL:

cd /usr/local/src/

wget https://www.openssl.org/source/openssl-1.1.1b.tar.gz && tar xzvf openssl-1.1.1b.tar.gz

cd /usr/local/src/openssl-1.1.1b

./config --prefix=/usr/local/openssl-1.1.1b --openssldir=/usr/local/openssl-1.1.1b shared zlib no-ssl3 no-weak-ssl-ciphers

make -j2 && make test
make install

export OPENSSL_LIB=/usr/local/openssl-1.1.1b/lib
export OPENSSL_INC=/usr/local/openssl-1.1.1b/include
export OPENSSL_DIRECTORY="/usr/local/src/openssl-1.1.1b"

# Setup PATH environment variables:
cat > /etc/profile.d/openssl.sh << __EOF__
#!/bin/sh
export PATH=/usr/local/openssl-1.1.1b/bin:${PATH}
export LD_LIBRARY_PATH=/usr/local/openssl-1.1.1b/lib:${LD_LIBRARY_PATH}
__EOF__

chmod +x /etc/profile.d/openssl.sh && source /etc/profile.d/openssl.sh

# To make the OpenSSL 1.1.1b version visible globally first:
mv /usr/bin/openssl /usr/bin/openssl-1.1.0g
ln -s /usr/local/openssl-1.1.1b/bin/openssl /usr/bin/openssl

cat > /etc/ld.so.conf.d/openssl.conf << __EOF__
/usr/local/openssl-1.1.1b/lib
__EOF__

LuaJIT:

# I recommend to use OpenResty's branch (openresty/luajit2) instead LuaJIT (LuaJIT/LuaJIT), but both installation methods are similar:
cd /usr/local/src/

# For originall LuaJIT:
#   git clone http://luajit.org/git/luajit-2.0
#   cd /usr/local/src/luajit-2.0

# For OpenResty's LuaJIT:
git clone --depth 1 https://github.com/openresty/luajit2

cd /usr/local/src/luajit2

make && make install

export LUA_LIB=/usr/local/lib/
export LUA_INC=/usr/local/include/luajit-2.1/

ln -s /usr/local/lib/libluajit-5.1.so.2.1.0 /usr/local/lib/liblua.so

sregex:

Required for replace-filter-nginx-module module.

cd /usr/local/src/

git clone --depth 1 https://github.com/openresty/sregex

cd /usr/local/src/sregex

make && make install

jemalloc:

To verify jemalloc in use: lsof -n | grep jemalloc.

cd /usr/local/src/

git clone --depth 1 https://github.com/jemalloc/jemalloc

cd /usr/local/src/jemalloc

./autogen.sh

make && make install

export JEMALLOC_DIRECTORY="/usr/local/src/jemalloc"

Update links and cache to the shared libraries for both types of installation:

ldconfig
Get Tengine sources
cd /usr/local/src/tengine

git clone --depth 1 https://github.com/alibaba/tengine master
Download 3rd party modules

Not all modules from this section working properly with Tengine (e.g. ndk_http_module and other dependent on it).

cd /usr/local/src/tengine/modules/

for i in \
https://github.com/openresty/echo-nginx-module \
https://github.com/openresty/headers-more-nginx-module \
https://github.com/openresty/replace-filter-nginx-module \
https://github.com/nginx-clojure/nginx-access-plus \
https://github.com/yaoweibin/ngx_http_substitutions_filter_module \
https://github.com/vozlt/nginx-module-vts \
https://github.com/google/ngx_brotli ; do

  git clone --depth 1 "$i"

done

wget http://mdounin.ru/hg/ngx_http_delay_module/archive/tip.tar.gz -O delay-module.tar.gz
mkdir delay-module && tar xzvf delay-module.tar.gz -C delay-module --strip 1

If you use NAXSI:

cd /usr/local/src/tengine/modules/

git clone --depth 1 https://github.com/nbs-system/naxsi
Build Tengine
cd /usr/local/src/tengine/master

# - you can also build Tengine without 3rd party modules
# - don't set values for --with-openssl, --with-pcre, and --with-zlib if you select prebuilt packages for them
./configure --prefix=/etc/nginx \
            --conf-path=/etc/nginx/nginx.conf \
            --sbin-path=/usr/sbin/nginx \
            --pid-path=/var/run/nginx.pid \
            --lock-path=/var/run/nginx.lock \
            --user=nginx \
            --group=nginx \
            --modules-path=/etc/nginx/modules \
            --error-log-path=/var/log/nginx/error.log \
            --http-log-path=/var/log/nginx/access.log \
            --http-client-body-temp-path=/var/cache/nginx/client_temp \
            --http-proxy-temp-path=/var/cache/nginx/proxy_temp \
            --http-fastcgi-temp-path=/var/cache/nginx/fastcgi_temp \
            --http-uwsgi-temp-path=/var/cache/nginx/uwsgi_temp \
            --http-scgi-temp-path=/var/cache/nginx/scgi_temp \
            --with-compat \
            --with-debug \
            --with-file-aio \
            --with-threads \
            --with-stream \
            --with-stream_geoip_module \
            --with-stream_realip_module \
            --with-stream_ssl_module \
            --with-stream_ssl_preread_module \
            --with-http_addition_module \
            --with-http_auth_request_module \
            --with-http_backtrace_module \
            --with-http_debug_pool_module \
            --with-http_debug_timer_module \
            --with-http_degradation_module \
            --with-http_footer_filter_module \
            --with-http_geoip_module \
            --with-http_gunzip_module \
            --with-http_gzip_static_module \
            --with-http_image_filter_module \
            --with-http_lua_module \
            --with-http_perl_module \
            --with-http_random_index_module \
            --with-http_realip_module \
            --with-http_secure_link_module \
            --with-http_ssl_module \
            --with-http_stub_status_module \
            --with-http_sub_module \
            --with-http_sysguard_module \
            --with-http_upstream_check_module \
            --with-http_upstream_session_sticky_module \
            --with-http_v2_module \
            --with-google_perftools_module \
            --with-openssl=${OPENSSL_DIRECTORY} \
            --with-openssl-opt=no-weak-ssl-ciphers \
            --with-openssl-opt=no-ssl3 \
            --with-pcre=${PCRE_DIRECTORY} \
            --with-pcre-jit \
            --with-jemalloc=${JEMALLOC_DIRECTORY} \
            --without-http-cache \
            --without-http_memcached_module \
            --without-mail_pop3_module \
            --without-mail_imap_module \
            --without-mail_smtp_module \
            --without-http_fastcgi_module \
            --without-http_scgi_module \
            --without-http_uwsgi_module \
            --with-cc-opt='-I/usr/local/include -I/usr/local/openssl-1.1.1b/include -I/usr/local/include/luajit-2.1/ -I/usr/local/include/jemalloc -O2 -g -pipe -Wall -Wp,-D_FORTIFY_SOURCE=2 -fexceptions -fstack-protector-strong --param=ssp-buffer-size=4 -grecord-gcc-switches -m64 -mtune=generic -fPIC' \
            --with-ld-opt='-Wl,-E -L/usr/local/lib -ljemalloc -lpcre -Wl,-rpath,/usr/local/lib/,-z,relro -Wl,-z,now -pie' \
            --add-module=/usr/local/src/tengine/modules/nginx-access-plus/src/c \
            --add-module=/usr/local/src/tengine/modules/ngx_http_substitutions_filter_module \
            --add-module=/usr/local/src/tengine/modules/nginx-module-vts \
            --add-module=/usr/local/src/tengine/modules/ngx_brotli \
            --add-dynamic-module=/usr/local/src/tengine/modules/echo-nginx-module \
            --add-dynamic-module=/usr/local/src/tengine/modules/headers-more-nginx-module \
            --add-dynamic-module=/usr/local/src/tengine/modules/replace-filter-nginx-module \
            --add-dynamic-module=/usr/local/src/tengine/modules/delay-module \
            --add-dynamic-module=/usr/local/src/tengine/modules/naxsi/naxsi_src

make -j2 && make test
make install

ldconfig

Check Tengine version:

nginx -v
Tengine version: Tengine/2.3.0
nginx version: nginx/1.15.9

And list all files in /etc/nginx:

tree
.
├── fastcgi.conf
├── fastcgi.conf.default
├── fastcgi_params
├── fastcgi_params.default
├── html
│   ├── 50x.html
│   └── index.html
├── koi-utf
├── koi-win
├── mime.types
├── mime.types.default
├── modules
│   ├── ngx_http_delay_module.so
│   ├── ngx_http_echo_module.so
│   ├── ngx_http_headers_more_filter_module.so
│   └── ngx_http_replace_filter_module.so
├── nginx.conf
├── nginx.conf.default
├── scgi_params
├── scgi_params.default
├── uwsgi_params
├── uwsgi_params.default
└── win-utf

2 directories, 21 files
Post installation tasks

Check all post installation tasks from post installation tasks - NGINX section.

Base Rules

These are the basic set of rules to keep NGINX in good condition.

🔰 Organising Nginx configuration

Rationale

When your NGINX configuration grow, the need for organising your configuration will also grow. Well organised code is:

  • easier to understand
  • easier to maintain
  • easier to work with

Use include directive to move common server settings into a separate files and to attach your NGINX specific code to global config, contexts and other.

Example
# Store this configuration in e.g. https-ssl-common.conf
listen 10.240.20.2:443 ssl;

root /etc/nginx/error-pages/other;

ssl_certificate /etc/nginx/domain.com/certs/nginx_domain.com_bundle.crt;
ssl_certificate_key /etc/nginx/domain.com/certs/domain.com.key;

# And include this file in server section:
server {

  include /etc/nginx/domain.com/commons/https-ssl-common.conf;

  server_name domain.com www.domain.com;

  ...
External resources

🔰 Format, prettify and indent your Nginx code

Rationale

Work with unreadable configuration files is terrible, if syntax isn’t very readable, it makes your eyes sore, and you suffers from headaches.

When your code is formatted, it is significantly easier to maintain, debug, optimize, and can be read and understood in a short amount of time. You should eliminate code style violations from your NGINX configuration files.

Choose your formatter style and setup a common config for it. Some rules are universal, but the most important thing is to keep a consistent NGINX code style throughout your code base:

  • use whitespaces and blank lines to arrange and separate code blocks
  • use tabs for indents - they are consistent, customizable and allow mistakes to be more noticeable (unless you are a 4 space kind of guy)
  • use comments to explain why things are done not what is done
  • use meaningful naming conventions
  • simple is better than complex but complex is better than complicated

Of course, the NGINX configuration code is not a programming language. All files are written in their own language or syntax so we should not overdo it, but I think it's worth sticking to the general rules and make your and other NGINX adminstrators life easier.

Example
# Good NGINX code style:
http {

  # Attach global rules:
  include         /etc/nginx/proxy.conf;
  include         /etc/nginx/fastcgi.conf;

  index           index.html index.htm index.php;

  default_type    application/octet-stream;

  # Standard log format:
  log_format      main '$remote_addr - $remote_user [$time_local]  $status '
                       '"$request" $body_bytes_sent "$http_referer" '
                       '"$http_user_agent" "$http_x_forwarded_for"';

  access_log      /var/log/nginx/access.log main;

  sendfile        on;
  tcp_nopush      on;

  # This seems to be required for some vhosts:
  server_names_hash_bucket_size 128;

  ...

# Bad NGINX code style:
http {
  include    nginx/proxy.conf;
  include    /etc/nginx/fastcgi.conf;
  index    index.html index.htm index.php;

  default_type application/octet-stream;
  log_format   main '$remote_addr - $remote_user [$time_local]  $status '
    '"$request" $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for"';
  access_log   logs/access.log    main;
  sendfile on;
  tcp_nopush   on;
  server_names_hash_bucket_size 128; # this seems to be required for some vhosts

  ...
External resources

🔰 Separate listen directives for 80 and 443

Rationale

I don't like duplicating the rules, but it's certainly an easy and maintainable way.

Example
# For http:
server {

  listen 10.240.20.2:80;

  ...

}

# For https:
server {

  listen 10.240.20.2:443 ssl;

  ...

}
External resources

🔰 Define the listen directives explicitly with address:port pair

Rationale

NGINX translates all incomplete listen directives by substituting missing values with their default values.

NGINX will only evaluate the server_name directive when it needs to distinguish between server blocks that match to the same level in the listen directive.

Set IP address and port number to prevents soft mistakes which may be difficult to debug.

Example
server {

  # This block will be processed:
  listen 192.168.252.10;  # --> 192.168.252.10:80

  ...

}

server {

  listen 80;  # --> *:80 --> 0.0.0.0:80
  server_name api.random.com;

  ...

}
External resources

🔰 Prevent processing requests with undefined server names

Rationale

NGINX should prevent processing requests with undefined server names (also on IP address). It also protects against configuration errors and don't pass traffic to incorrect backends. The problem is easily solved by creating a default catch all server config.

If none of the listen directives have the default_server parameter then the first server with the address:port pair will be the default server for this pair (it means that NGINX always has a default server).

If someone makes a request using an IP address instead of a server name, the Host request header field will contain the IP address and the request can be handled using the IP address as the server name.

Also good point is return 444; for default server name because this will close the connection and log it internally, for any domain that isn't defined in NGINX.

Example
# Place it at the beginning of the configuration file to prevent mistakes.
server {

  # Add default_server to your listen directive in the server that you want to act as the default.
  listen 10.240.20.2:443 default_server ssl;

  # We catch:
  #   - invalid domain names
  #   - requests without the "Host" header
  #   - and all others (also due to the above setting)
  #   - default_server in server_name directive is not required - I add this for a better understanding
  server_name _ "" default_server;

  ...

  return 444;

  # We can also serve:
  # location / {

    # static file (error page):
    # root /etc/nginx/error-pages/404;
    # or redirect:
    # return 301 https://badssl.com;

    # return 444;

  # }

}

server {

  listen 10.240.20.2:443 ssl;

  server_name domain.com;

  ...

}

server {

  listen 10.240.20.2:443 ssl;

  server_name domain.org;

  ...

}
External resources

🔰 Use reload method to change configurations on the fly

Rationale

Use the reload method of NGINX to achieve a graceful reload of the configuration without stopping the server and dropping any packets.

This ability of NGINX is very critical in a high-uptime, dynamic environments for keeping the load balancer or standalone server online.

When you restart NGINX you might encounter situation in which NGINX will stop, and won't start back again, because of syntax error. Reload method is safer than restarting because before old process will be terminated, new configuration file is parsed and whole process is aborted if there are any problems with it.

Example
# 1)
systemctl reload nginx

# 2)
service nginx reload

# 3)
/etc/init.d/nginx reload

# 4)
/usr/sbin/nginx -s reload

# 5)
kill -HUP $(cat /var/run/nginx.pid)
# or
kill -HUP $(ps auxw | grep [n]ginx | grep master | awk '{print $2}')
External resources

🔰 Use only one SSL config for specific listen directive

Rationale

For sharing a single IP address between several HTTPS servers you should use one SSL config (e.g. protocols, ciphers, curves) because changes will affect only the default server.

Remember that regardless of SSL parameters, you are able to use multiple SSL certificates.

If you want to set up different SSL configurations for the same IP address then it will fail. It's important because SSL configuration is presented for default server - if none of the listen directives have the default_server parameter then the first server in your configuration. So you should use only one SSL setup with several names on the same IP address.

It's also to prevent mistakes and configuration mismatch.

Example
# Store this configuration in e.g. https.conf
listen 192.168.252.10:443 default_server ssl http2;

ssl_protocols TLSv1.2;
ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384";

ssl_prefer_server_ciphers on;

ssl_ecdh_curve secp521r1:secp384r1;

...

# Include this file to the server context (attach domain-a.com for specific listen directive)
server {

  include /etc/nginx/https.conf;

  server_name domain-a.com;

  ...

}

# Include this file to the server context (attach domain-b.com for specific listen directive)
server {

  include /etc/nginx/https.conf;

  server_name domain-b.com;

  ...

}
External resources

🔰 Force all connections over TLS

Rationale

You should always use HTTPS instead of HTTP to protect your website, even if it doesn’t handle sensitive communications.

We have currently the first free and open CA - Let's Encrypt - so generating and implementing certificates has never been so easy. It was created to provide free and easy-to-use TLS and SSL certificates.

Example
server {

  listen 10.240.20.2:80;

  server_name domain.com;

  return 301 https://$host$request_uri;

}

server {

  listen 10.240.20.2:443 ssl;

  server_name domain.com;

  ...

}
External resources

🔰 Use geo/map modules instead allow/deny

Rationale

Use map or geo modules (one of them) to prevent users abusing your servers.

This allows to create variables with values depending on the client IP address.

Example
# Map module:
map $remote_addr $globals_internal_map_acl {

  # Status code:
  #  - 0 = false
  #  - 1 = true
  default 0;

  ### INTERNAL ###
  10.255.10.0/24 1;
  10.255.20.0/24 1;
  10.255.30.0/24 1;
  192.168.0.0/16 1;

}

# Geo module:
geo $globals_internal_geo_acl {

  # Status code:
  #  - 0 = false
  #  - 1 = true
  default 0;

  ### INTERNAL ###
  10.255.10.0/24 1;
  10.255.20.0/24 1;
  10.255.30.0/24 1;
  192.168.0.0/16 1;

}
External resources

🔰 Map all the things...

Rationale

Manage a large number of redirects with NGINX maps and use them to customize your key-value pairs.

Map module provides a more elegant solution for clearly parsing a big list of regexes, e.g. User-Agents, Referrers.

Example
map $http_user_agent $device_redirect {

  default "desktop";

  ~(?i)ip(hone|od) "mobile";
  ~(?i)android.*(mobile|mini) "mobile";
  ~Mobile.+Firefox "mobile";
  ~^HTC "mobile";
  ~Fennec "mobile";
  ~IEMobile "mobile";
  ~BB10 "mobile";
  ~SymbianOS.*AppleWebKit "mobile";
  ~Opera\sMobi "mobile";

}

# Turn on in a specific context (e.g. location):
if ($device_redirect = "mobile") {

  return 301 https://m.domain.com$request_uri;

}
External resources

🔰 Drop the same root inside location block

Rationale

If you add a root to every location block then a location block that isn’t matched will have no root. Set global root inside server directive.

Example
server {

  server_name domain.com;

  root /var/www/domain.com/public;

  location / {

    ...

  }

  location /api {

    ...

  }

  location /static {

    root /var/www/domain.com/static;

    ...

  }

}
External resources

🔰 Use debug mode for debugging

Rationale

There's probably more detail than you want, but that can sometimes be a lifesaver (but log file growing rapidly on a very high-traffic sites).

Example
rewrite_log on;
error_log /var/log/nginx/error-debug.log debug;
External resources

🔰 Use custom log formats for debugging

Rationale

Anything you can access as a variable in NGINX config, you can log, including non-standard http headers, etc. so it's a simple way to create your own log format for specific situations.

This is extremely helpful for debugging specific location directives.

Example
# Default main log format from NGINX repository:
log_format main
                '$remote_addr - $remote_user [$time_local] "$request" '
                '$status $body_bytes_sent "$http_referer" '
                '"$http_user_agent" "$http_x_forwarded_for"';

# Extended main log format:
log_format main-level-0
                '$remote_addr - $remote_user [$time_local] '
                '"$request_method $scheme://$host$request_uri '
                '$server_protocol" $status $body_bytes_sent '
                '"$http_referer" "$http_user_agent" '
                '$request_time';

# Debug log formats:
log_format debug-level-0
                '$remote_addr - $remote_user [$time_local] '
                '"$request_method $scheme://$host$request_uri '
                '$server_protocol" $status $body_bytes_sent '
                '$request_id $pid $msec $request_time '
                '$upstream_connect_time $upstream_header_time '
                '$upstream_response_time "$request_filename" '
                '$request_completion';

log_format debug-level-1
                '$remote_addr - $remote_user [$time_local] '
                '"$request_method $scheme://$host$request_uri '
                '$server_protocol" $status $body_bytes_sent '
                '$request_id $pid $msec $request_time '
                '$upstream_connect_time $upstream_header_time '
                '$upstream_response_time "$request_filename" $request_length '
                '$request_completion $connection $connection_requests';

log_format debug-level-2
                '$remote_addr - $remote_user [$time_local] '
                '"$request_method $scheme://$host$request_uri '
                '$server_protocol" $status $body_bytes_sent '
                '$request_id $pid $msec $request_time '
                '$upstream_connect_time $upstream_header_time '
                '$upstream_response_time "$request_filename" $request_length '
                '$request_completion $connection $connection_requests '
                '$remote_addr $remote_port $server_addr $server_port';
External resources

Performance

NGINX is a insanely fast, but you can adjust a few things to make sure it's as fast as possible for your use case.

🔰 Adjust worker processes

Rationale

The worker_processes directive is the sturdy spine of life for NGINX. This directive is responsible for letting our virtual server know many workers to spawn once it has become bound to the proper IP and port(s).

I think for high load proxy servers (also standalone servers) good value is ALL_CORES - 1 (please test it before used).

Rule of thumb: If much time is spent blocked on I/O, worker processes should be increased further.

Official NGINX documentation say:

When one is in doubt, setting it to the number of available CPU cores would be a good start (the value "auto" will try to autodetect it).

Example
# VCPU = 4 , expr $(nproc --all) - 1
worker_processes 3;
External resources

🔰 Use HTTP/2

Rationale

The primary goals for HTTP/2 are to reduce latency by enabling full request and response multiplexing, minimize protocol overhead via efficient compression of HTTP header fields, and add support for request prioritization and server push.

HTTP/2 will make our applications faster, simpler, and more robust.

HTTP/2 is backwards-compatible with HTTP/1.1, so it would be possible to ignore it completely and everything will continue to work as before because if the client that does not support HTTP/2 will never ask the server for an HTTP/2 communication upgrade: the communication between them will be fully HTTP1/1.

Example
# For https:
server {

  listen 10.240.20.2:443 ssl http2;

  ...
External resources

🔰 Maintaining SSL sessions

Rationale

This improves performance from the clients’ perspective, because it eliminates the need for a new (and time-consuming) SSL handshake to be conducted each time a request is made.

Most servers do not purge sessions or ticket keys, thus increasing the risk that a server compromise would leak data from previous (and future) connections.

Set SSL Session Timeout to 5 minutes for prevent abused by advertisers like Google and Facebook.

Example
ssl_session_cache shared:SSL:10m;
ssl_session_timeout 5m;
ssl_session_tickets off;
ssl_buffer_size 1400;
External resources

🔰 Use exact names in server_name directive where possible

Rationale

Exact names, wildcard names starting with an asterisk, and wildcard names ending with an asterisk are stored in three hash tables bound to the listen ports.

The exact names hash table is searched first. If a name is not found, the hash table with wildcard names starting with an asterisk is searched. If the name is not found there, the hash table with wildcard names ending with an asterisk is searched. Searching wildcard names hash table is slower than searching exact names hash table because names are searched by domain parts.

Regular expressions are tested sequentially and therefore are the slowest method and are non-scalable. For these reasons, it is better to use exact names where possible.

Example
# It is more efficient to define them explicitly:
server {

    listen       80;

    server_name  example.org  www.example.org  *.example.org;

    ...

}

# than to use the simplified form:
server {

    listen       80;

    server_name  .example.org;

    ...

}
External resources

🔰 Avoid checks server_name with if directive

Rationale

When NGINX receives a request no matter what is the subdomain being requested, be it www.example.com or just the plain example.com this if directive is always evaluated. Since you’re requesting NGINX to check for the Host header for every request. It’s extremely inefficient.

Instead use two server directives like the example below. This approach decreases NGINX processing requirements.

Example
# Bad configuration:
server {

  ...

  server_name                 domain.com www.domain.com;

  if ($host = www.domain.com) {

    return                    301 https://domain.com$request_uri;

  }

  server_name                 domain.com;

  ...

}

# Good configuration:
server {

    server_name               www.domain.com;
    return                    301 $scheme://domain.com$request_uri;
    # If you force your web traffic to use HTTPS:
    #                         301 https://domain.com$request_uri;

}

server {

    listen                    80;

    server_name               domain.com;

    ...

}
External resources

🔰 Make an exact location match to speed up the selection process

Rationale

Exact location matches are often used to speed up the selection process by immediately ending the execution of the algorithm.

Example
# Matches the query / only and stops searching:
location = / {

  ...

}

# Matches the query /v9 only and stops searching:
location = /v9 {

  ...

}

...

# Matches any query due to the fact that all queries begin at /,
# but regular expressions and any longer conventional blocks will be matched at first place:
location / {

  ...

}
External resources

🔰 Use limit_conn to improve limiting the download speed

Rationale

NGINX provides two directives to limiting download speed:

  • limit_rate_after - sets the amount of data transferred before the limit_rate directive takes effect
  • limit_rate - allows you to limit the transfer rate of individual client connections (past exceeding limit_rate_after)

This solution limits NGINX download speed per connection, so, if one user opens multiple e.g. video files, it will be able to download X * the number of times he connected to the video files.

To prevent this situation use limit_conn_zone and limit_conn directives.

Example
# Create limit connection zone:
limit_conn_zone $binary_remote_addr zone=conn_for_remote_addr:1m;

# Add rules to limiting the download speed:
limit_rate_after 1m;  # run at maximum speed for the first 1 megabyte
limit_rate 250k;      # and set rate limit after 1 megabyte

# Enable queue:
location /videos {

  # Max amount of data by one client: 10 megabytes (limit_rate_after * 10)
  limit_conn conn_for_remote_addr 10;

  ...
External resources

Hardening

In this chapter I will talk about some of the NGINX hardening approaches and security standards.

🔰 Run as an unprivileged user

Rationale

There is no real difference in security just by changing the process owner name. On the other hand in security, the principle of least privilege states that an entity should be given no more permission than necessary to accomplish its goals within a given system. This way only master process runs as root.

Example
# Edit nginx.conf:
user nginx;

# Set owner and group for root (app, default) directory:
chown -R nginx:nginx /var/www/domain.com
External resources

🔰 Disable unnecessary modules

Rationale

It is recommended to disable any modules which are not required as this will minimize the risk of any potential attacks by limiting the operations allowed by the web server.

The best way to disable unused modules you should use the configure option during installation.

Example
# During installation:
./configure --without-http_autoindex_module

# Comment modules in the configuration file e.g. modules.conf:
# load_module                 /usr/share/nginx/modules/ndk_http_module.so;
# load_module                 /usr/share/nginx/modules/ngx_http_auth_pam_module.so;
# load_module                 /usr/share/nginx/modules/ngx_http_cache_purge_module.so;
# load_module                 /usr/share/nginx/modules/ngx_http_dav_ext_module.so;
load_module                   /usr/share/nginx/modules/ngx_http_echo_module.so;
# load_module                 /usr/share/nginx/modules/ngx_http_fancyindex_module.so;
load_module                   /usr/share/nginx/modules/ngx_http_geoip_module.so;
load_module                   /usr/share/nginx/modules/ngx_http_headers_more_filter_module.so;
# load_module                 /usr/share/nginx/modules/ngx_http_image_filter_module.so;
# load_module                 /usr/share/nginx/modules/ngx_http_lua_module.so;
load_module                   /usr/share/nginx/modules/ngx_http_perl_module.so;
# load_module                 /usr/share/nginx/modules/ngx_mail_module.so;
# load_module                 /usr/share/nginx/modules/ngx_nchan_module.so;
# load_module                 /usr/share/nginx/modules/ngx_stream_module.so;
External resources

🔰 Protect sensitive resources

Rationale

Hidden directories and files should never be web accessible - sometimes critical data are published during application deploy. If you use control version system you should defninitely drop the access to the critical hidden directories like a .git or .svn to prevent expose source code of your application.

Sensitive resources contains items that abusers can use to fully recreate the source code used by the site and look for bugs, vulnerabilities, and exposed passwords.

Example
if ($request_uri ~ "/\.git") {

  return 403;

}

# or
location ~ /\.git {

  deny all;

}

# or
location ~* ^.*(\.(?:git|svn|htaccess))$ {

  return 403;

}

# or all . directories/files excepted .well-known
location ~ /\.(?!well-known\/) {

  deny all;

}
External resources

🔰 Hide Nginx version number

Rationale

Disclosing the version of NGINX running can be undesirable, particularly in environments sensitive to information disclosure.

But the "Official Apache Documentation (Apache Core Features)" say:

Setting ServerTokens to less than minimal is not recommended because it makes it more difficult to debug interoperational problems. Also note that disabling the Server: header does nothing at all to make your server more secure. The idea of "security through obscurity" is a myth and leads to a false sense of safety.

Example
server_tokens off;
External resources

🔰 Hide Nginx server signature

Rationale

In my opinion there is no real reason or need to show this much information about your server. It is easy to look up particular vulnerabilities once you know the version number.

You should compile NGINX from sources with ngx_headers_more to used more_set_headers directive.

Example
more_set_headers "Server: Unknown";
External resources

🔰 Hide upstream proxy headers

Rationale

When NGINX is used to proxy requests to an upstream server (such as a PHP-FPM instance), it can be beneficial to hide certain headers sent in the upstream response (e.g. the version of PHP running).

Example
proxy_hide_header X-Powered-By;
proxy_hide_header X-AspNetMvc-Version;
proxy_hide_header X-AspNet-Version;
proxy_hide_header X-Drupal-Cache;
External resources

🔰 Use only the latest supported OpenSSL version

Rationale

Before start see Release Strategy Policies and Changelog on the OpenSSL website.

Criteria for choosing OpenSSL version can vary and it depends all on your use.

The latest versions of the major OpenSSL library are (may be changed):

  • the next version of OpenSSL will be 3.0.0
  • version 1.1.1 will be supported until 2023-09-11 (LTS)
    • last minor version: 1.1.1b (February 26, 2019)
  • version 1.1.0 will be supported until 2019-09-11
    • last minor version: 1.1.0j (November 20, 2018)
  • version 1.0.2 will be supported until 2019-12-31 (LTS)
    • last minor version: 1.0.2r (February 26, 2019)
  • any other versions are no longer supported

In my opinion the only safe way is based on the up-to-date and still supported version of the OpenSSL. And what's more, I recommend to hang on to the latest versions (e.g. 1.1.1).

If your system repositories do not have the newest OpenSSL, you can do the compilation process (see OpenSSL sub-section).

External resources

🔰 Use min. 2048-bit private keys

Rationale

Advisories recommend 2048 for now. Security experts are projecting that 2048 bits will be sufficient for commercial use until around the year 2030 (as per NIST).

The latest version of FIPS-186 also say the U.S. Federal Government generate (and use) digital signatures with 1024, 2048, or 3072 bit key lengths.

Generally there is no compelling reason to choose 4096 bit keys over 2048 provided you use sane expiration intervals.

If you want to get A+ with 100%s on SSL Lab (for Key Exchange) you should definitely use 4096 bit private keys. That's the main reason why you should use them.

Longer keys take more time to generate and require more CPU (please use openssl speed rsa on your server) and power when used for encrypting and decrypting, also the SSL handshake at the start of each connection will be slower. It also has a small impact on the client side (e.g. browsers).

Use of alternative solution: ECC Certificate Signing Request (CSR) - ECDSA certificates contain an ECC public key. ECC keys are better than RSA & DSA keys in that the ECC algorithm is harder to break.

The "SSL/TLS Deployment Best Practices" book say:

The cryptographic handshake, which is used to establish secure connections, is an operation whose cost is highly influenced by private key size. Using a key that is too short is insecure, but using a key that is too long will result in "too much" security and slow operation. For most web sites, using RSA keys stronger than 2048 bits and ECDSA keys stronger than 256 bits is a waste of CPU power and might impair user experience. Similarly, there is little benefit to increasing the strength of the ephemeral key exchange beyond 2048 bits for DHE and 256 bits for ECDHE.

Konstantin Ryabitsev (Reddit):

Generally speaking, if we ever find ourselves in a world where 2048-bit keys are no longer good enough, it won't be because of improvements in brute-force capabilities of current computers, but because RSA will be made obsolete as a technology due to revolutionary computing advances. If that ever happens, 3072 or 4096 bits won't make much of a difference anyway. This is why anything above 2048 bits is generally regarded as a sort of feel-good hedging theatre.

My recommendation:

Use 2048-bit key instead 4096-bit at this moment.

Example
### Example (RSA):
( _fd="domain.com.key" ; _len="2048" ; openssl genrsa -out ${_fd} ${_len} )

# Let's Encrypt:
certbot certonly -d domain.com -d www.domain.com --rsa-key-size 2048

### Example (ECC):
# _curve: prime256v1, secp521r1, secp384r1
( _fd="domain.com.key" ; _fd_csr="domain.com.csr" ; _curve="prime256v1" ; \
openssl ecparam -out ${_fd} -name ${_curve} -genkey ; \
openssl req -new -key ${_fd} -out ${_fd_csr} -sha256 )

# Let's Encrypt (from above):
certbot --csr ${_fd_csr} -[other-args]

For x25519:

( _fd="private.key" ; _curve="x25519" ; \
openssl genpkey -algorithm ${_curve} -out ${_fd} )

  arrowtr_100

( _fd="domain.com.key" ; _len="2048" ; openssl genrsa -out ${_fd} ${_len} )

# Let's Encrypt:
certbot certonly -d domain.com -d www.domain.com

  arrowtr_90

External resources

🔰 Keep only TLS 1.2 and TLS 1.3

Rationale

It is recommended to run TLS 1.1/1.2/1.3 and fully disable SSLv2, SSLv3 and TLS 1.0 that have protocol weaknesses.

TLS 1.1 and 1.2 are both without security issues - but only TLS 1.2 and TLS 1.3 provides modern cryptographic algorithms. TLS 1.3 is a new TLS version that will power a faster and more secure web for the next few years. TLS 1.0 and TLS 1.1 protocols will be removed from browsers at the beginning of 2020.

TLS 1.2 does require careful configuration to ensure obsolete cipher suites with identified vulnerabilities are not used in conjunction with it. TLS 1.3 removes the need to make these decisions. TLS 1.3 version also improves TLS 1.2 security, privace and performance issues.

Before enabling specific protocol version, you should check which ciphers are supported by the protocol. So if you turn on TLS 1.1, TLS 1.2 and TLS 1.3 both remember about the correct (and strong) ciphers to handle them. Otherwise, they will not be anyway works without supported ciphers (no TLS handshake will succeed).

If you told NGINX to use TLS 1.3, it will use TLS 1.3 only where is available. NGINX supports TLS 1.3 since version 1.13.0 (released in April 2017), when built against OpenSSL 1.1.1 or more.

My recommendation:

Use only TLSv1.3 and TLSv1.2.

Example

TLS 1.3 + 1.2:

ssl_protocols TLSv1.3 TLSv1.2;

TLS 1.2:

ssl_protocols TLSv1.2;

  arrowtr_100

TLS 1.3 + 1.2 + 1.1:

ssl_protocols TLSv1.3 TLSv1.2 TLSv1.1;

TLS 1.2 + 1.1:

ssl_protocols TLSv1.2 TLSv1.1;

  arrowtr_95

External resources

🔰 Use only strong ciphers

Rationale

This parameter changes quite often, the recommended configuration for today may be out of date tomorrow.

To check ciphers supported by OpenSSL on your server: openssl ciphers -s -v, openssl ciphers -s -v ECDHE or openssl ciphers -s -v DHE.

For more security use only strong and not vulnerable cipher suites. Place ECDHE and DHE suites at the top of your list. The order is important; because ECDHE suites are faster, you want to use them whenever clients supports them.

For backward compatibility software components you should use less restrictive ciphers. Not only that you have to enable at least one special AES128 cipher for HTTP/2 support regarding to RFC7540: TLS 1.2 Cipher Suites, you also have to allow prime256 elliptic curves which reduces the score for key exchange by another 10% even if a secure server preferred order is set.

Also modern cipher suites (e.g. from Mozilla recommendations) suffers from compatibility troubles mainly because drops SHA-1. But be careful if you want to use ciphers with HMAC-SHA-1 - there's a perfectly good explanation why.

If you want to get A+ with 100%s on SSL Lab (for Cipher Strength) you should definitely disable 128-bit ciphers. That's the main reason why you should not use them.

In my opinion 128-bit symmetric encryption doesn’t less secure. For example TLS 1.3 use TLS_AES_128_GCM_SHA256 (0x1301) (for TLS-compliant applications). It is not possible to control ciphers for TLS 1.3 without support from client to use new API for TLSv1.3 cipher suites so at this moment it's always on (also if you disable potentially weak cipher from NGINX). On the other hand the ciphers in TLSv1.3 have been restricted to only a handful of completely secure ciphers by leading crypto experts.

For TLS 1.2 you should consider disable weak ciphers without forward secrecy like ciphers with CBC algorithm. Using them also reduces the final grade because they don't use ephemeral keys. In my opinion you should use ciphers with AEAD (TLS 1.3 supports only these suites) encryption because they don't have any known weaknesses.

You should also absolutely disable weak ciphers regardless of the TLS version do you use, like those with DSS, DSA, DES/3DES, RC4, MD5, SHA1, null, anon in the name.

We have a nice online tool for testing compatibility cipher suites with user agents: CryptCheck. I think it will be very helpful for you.

My recommendation:

Use only TLSv1.3 and TLSv1.2 with below cipher suites:

ssl_ciphers "TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256";
Example

Cipher suites for TLS 1.3:

ssl_ciphers "TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-256-GCM-SHA384";

Cipher suites for TLS 1.2:

ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384";

  arrowtr_100

Cipher suites for TLS 1.3:

ssl_ciphers "TLS13-CHACHA20-POLY1305-SHA256:TLS13-AES-256-GCM-SHA384:TLS13-AES-128-GCM-SHA256";

Cipher suites for TLS 1.2:

# 1)
ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-SHA384";

# 2)
ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256";

# 3)
ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256";

# 4)
ssl_ciphers "EECDH+CHACHA20:EDH+AESGCM:AES256+EECDH:AES256+EDH";

Cipher suites for TLS 1.1 + 1.2:

# 1)
ssl_ciphers "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES128-GCM-SHA256";

# 2)
ssl_ciphers "ECDHE-ECDSA-CHACHA20-POLY1305:ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:!AES256-GCM-SHA256:!AES256-GCM-SHA128:!aNULL:!MD5";

  arrowtr_90

This will also give a baseline for comparison with Mozilla SSL Configuration Generator:

  • Modern profile with OpenSSL 1.1.0b (TLSv1.2)
ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
  • Intermediate profile with OpenSSL 1.1.0b (TLSv1, TLSv1.1 and TLSv1.2)
ssl_ciphers 'ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS';
External resources

🔰 Use more secure ECDH Curve

Rationale

For a SSL server certificate, an "elliptic curve" certificate will be used only with digital signatures (ECDSA algorithm).

x25519 is a more secure but slightly less compatible option. To maximise interoperability with existing browsers and servers, stick to P-256 prime256v1 and P-384 secp384r1 curves.

NSA Suite B says that NSA uses curves P-256 and P-384 (in OpenSSL, they are designated as, respectively, prime256v1 and secp384r1). There is nothing wrong with P-521, except that it is, in practice, useless. Arguably, P-384 is also useless, because the more efficient P-256 curve already provides security that cannot be broken through accumulation of computing power.

Use P-256 to minimize trouble. If you feel that your manhood is threatened by using a 256-bit curve where a 384-bit curve is available, then use P-384: it will increases your computational and network costs.

If you use TLS 1.3 you should enable prime256v1 signature algorithm. Without this SSL Lab reports TLS_AES_128_GCM_SHA256 (0x1301) signature as weak.

If you do not set ssh_ecdh_curve, then the NGINX will use its default settings, e.g. Chrome will prefer x25519, but this is not recommended because you can not control the NGINX's default settings (seems to be P-256).

Explicitly set ssh_ecdh_curve X25519:prime256v1:secp521r1:secp384r1; decreases the Key Exchange SSL Labs rating.

Definitely do not use the secp112r1, secp112r2, secp128r1, secp128r2, secp160k1, secp160r1, secp160r2, secp192k1 curves. They have a too small size for security application according to NIST recommendation.

My recommendation:

Use only TLSv1.3 and TLSv1.2 and only strong ciphers with above curves:

ssl_ecdh_curve X25519:secp521r1:secp384r1:prime256v1;
Example

Curves for TLS 1.2:

ssl_ecdh_curve secp521r1:secp384r1:prime256v1;

  arrowtr_100

# Alternative (this one doesn’t affect compatibility, by the way; it’s just a question of the preferred order).

# This setup downgrade Key Exchange score but is recommended for TLS 1.2 + 1.3:
ssl_ecdh_curve X25519:secp521r1:secp384r1:prime256v1;
External resources

🔰 Use strong Key Exchange

Rationale

The DH key is only used if DH ciphers are used. Modern clients prefer ECDHE instead and if your NGINX accepts this preference then the handshake will not use the DH param at all since it will not do a DHE key exchange but an ECDHE key exchange.

Most of the modern profiles from places like Mozilla's ssl config generator no longer recommend using this.

Default key size in OpenSSL is 1024 bits - it's vulnerable and breakable. For the best security configuration use your own 4096 bit DH Group or use known safe ones pre-defined DH groups (it's recommended) from mozilla.

Example
# To generate a DH key:
openssl dhparam -out /etc/nginx/ssl/dhparam_4096.pem 4096

# To produce "DSA-like" DH parameters:
openssl dhparam -dsaparam -out /etc/nginx/ssl/dhparam_4096.pem 4096

# To generate a ECDH key:
openssl ecparam -out /etc/nginx/ssl/ecparam.pem -name prime256v1

# NGINX configuration:
ssl_dhparam /etc/nginx/ssl/dhparams_4096.pem;

  arrowtr_100

External resources

🔰 Defend against the BEAST attack

Rationale

Generally the BEAST attack relies on a weakness in the way CBC mode is used in SSL/TLS.

More specifically, to successfully perform the BEAST attack, there are some conditions which needs to be met:

  • vulnerable version of SSL must be used using a block cipher (CBC in particular)
  • JavaScript or a Java applet injection - should be in the same origin of the web site
  • data sniffing of the network connection must be possible

To prevent possible use BEAST attacks you should enable server-side protection, which causes the server ciphers should be preferred over the client ciphers, and completely excluded TLS 1.0 from your protocol stack.

Example
ssl_prefer_server_ciphers on;
External resources

🔰 Disable HTTP compression or compress only zero sensitive content (mitigation of CRIME/BREACH attacks)

Rationale

You should probably never use TLS compression. Some user agents (at least Chrome) will disable it anyways. Disabling SSL/TLS compression stops the attack very effectively. A deployment of HTTP/2 over TLS 1.2 must disable TLS compression (please see RFC 7540: 9.2. Use of TLS Features).

CRIME exploits SSL/TLS compression which is disabled since nginx 1.3.2. BREACH exploits HTTP compression

Some attacks are possible (e.g. the real BREACH attack is a complicated) because of gzip (HTTP compression not TLS compression) being enabled on SSL requests. In most cases, the best action is to simply disable gzip for SSL.

Compression is not the only requirement for the attack to be done so using it does not mean that the attack will succeed. Generally you should consider whether having an accidental performance drop on HTTPS sites is better than HTTPS sites being accidentally vulnerable.

You shouldn't use HTTP compression on private responses when using TLS.

I would gonna to prioritize security over performance but compression can be (I think) okay to HTTP compress publicly available static content like css or js and HTML content with zero sensitive info (like an "About Us" page).

Remember: by default, NGINX doesn't compress image files using its per-request gzip module.

Gzip static module is better, for 2 reasons:

  • you don't have to gzip for each request
  • you can use a higher gzip level

You should put the gzip_static on; inside the blocks that configure static files, but if you’re only running one site, it’s safe to just put it in the http block.

Example
# Disable dynamic HTTP compression:
gzip off;

# Enable dynamic HTTP compression for specific location context:
location / {

  gzip on;

  ...

}

# Enable static gzip compression:
location ^~ /assets/ {

  gzip_static on;

  ...

}
External resources

🔰 HTTP Strict Transport Security

Rationale

The header indicates for how long a browser should unconditionally refuse to take part in unsecured HTTP connection for a specific domain.

Example
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains" always;

  arrowtr_A+

External resources

🔰 Reduce XSS risks (Content-Security-Policy)

Rationale

CSP reduce the risk and impact of XSS attacks in modern browsers.

Whitelisting known-good resource origins, refusing to execute potentially dangerous inline scripts, and banning the use of eval are all effective mechanisms for mitigating cross-site scripting attacks.

CSP is a good defence-in-depth measure to make exploitation of an accidental lapse in that less likely.

Before enable this header you should discuss with developers about it. They probably going to have to update your application to remove any inline script and style, and make some additional modifications there.

Example
# This policy allows images, scripts, AJAX, and CSS from the same origin, and does not allow any other resources to load.
add_header Content-Security-Policy "default-src 'none'; script-src 'self'; connect-src 'self'; img-src 'self'; style-src 'self';" always;
External resources

🔰 Control the behavior of the Referer header (Referrer-Policy)

Rationale

Determine what information is sent along with the requests.

Example
add_header Referrer-Policy "no-referrer";
External resources

🔰 Provide clickjacking protection (X-Frame-Options)

Rationale

Helps to protect your visitors against clickjacking attacks. It is recommended that you use the x-frame-options header on pages which should not be allowed to render a page in a frame.

Example
add_header X-Frame-Options "SAMEORIGIN" always;
External resources

🔰 Prevent some categories of XSS attacks (X-XSS-Protection)

Rationale

Enable the cross-site scripting (XSS) filter built into modern web browsers.

Example
add_header X-XSS-Protection "1; mode=block" always;
External resources

🔰 Prevent Sniff Mimetype middleware (X-Content-Type-Options)

Rationale

It prevents the browser from doing MIME-type sniffing (prevents "mime" based attacks).

Example
add_header X-Content-Type-Options "nosniff" always;
External resources

🔰 Deny the use of browser features (Feature-Policy)

Rationale

This header protects your site from third parties using APIs that have security and privacy implications, and also from your own team adding outdated APIs or poorly optimized images.

Example
add_header Feature-Policy "geolocation 'none'; midi 'none'; notifications 'none'; push 'none'; sync-xhr 'none'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; speaker 'none'; vibrate 'none'; fullscreen 'none'; payment 'none'; usb 'none';";
External resources

🔰 Reject unsafe HTTP methods

Rationale

Set of methods support by a resource. An ordinary web server supports the HEAD, GET and POST methods to retrieve static and dynamic content. Other (e.g. OPTIONS, TRACE) methods should not be supported on public web servers, as they increase the attack surface.

Example
add_header Allow "GET, POST, HEAD" always;

if ($request_method !~ ^(GET|POST|HEAD)$) {

  return 405;

}
External resources

🔰 Control Buffer Overflow attacks

Rationale

Buffer overflow attacks are made possible by writing data to a buffer and exceeding that buffers’ boundary and overwriting memory fragments of a process. To prevent this in NGINX we can set buffer size limitations for all clients.

Example
client_body_buffer_size 100k;
client_header_buffer_size 1k;
client_max_body_size 100k;
large_client_header_buffers 2 1k;
External resources

🔰 Mitigating Slow HTTP DoS attack (Closing Slow Connections)

Rationale

Close connections that are writing data too infrequently, which can represent an attempt to keep connections open as long as possible.

Example
client_body_timeout 10s;
client_header_timeout 10s;
keepalive_timeout 5s 5s;
send_timeout 10s;
External resources

Load Balancing

Load balancing is a useful mechanism to distribute incoming traffic around several capable servers. We may improve of some rules about the NGINX working as a load balancer.

🔰 Tweak passive health checks

Rationale

Monitoring for health is important on all types of load balancing mainly for business continuity. Passive checks watches for failed or timed-out connections as they pass through NGINX as requested by a client.

This functionality is enabled by default but the parameters mentioned here allow you to tweak their behavior. Default values are: max_fails=1 and fail_timeout=10s.

Example
upstream backend {

  server bk01_node:80 max_fails=3 fail_timeout=5s;
  server bk02_node:80 max_fails=3 fail_timeout=5s;

}
External resources

🔰 Don't disable backends by comments, use down parameter

Rationale

Sometimes we need to turn off backends e.g. at maintenance-time. I think good solution is marks the server as permanently unavailable with down parameter even if the downtime takes a short time.

Comments are good for really permanently disable servers or if you want to leave information for historical purposes.

NGINX also provides a backup parameter which marks the server as a backup server. It will be passed requests when the primary servers are unavailable. I use this option rarely for the above purposes and only if I am sure that the backends will work at the maintenance time.

Example
upstream backend {

  server bk01_node:80 max_fails=3 fail_timeout=5s down;
  server bk02_node:80 max_fails=3 fail_timeout=5s;

}
External resources

Others

This rules aren't strictly related to the NGINX but in my opinion they're also very important aspect of security.

🔰 Enable DNS CAA Policy

Rationale

DNS CAA policy helps you to control which Certificat Authorities are allowed to issue certificates for your domain becaues if no CAA record is present, any CA is allowed to issue a certificate for the domain.

Example

Generic configuration (Google Cloud DNS, Route 53, OVH, and other hosted services) for Let's Encrypt:

example.com. CAA 0 issue "letsencrypt.org"

Standard Zone File (BIND, PowerDNS and Knot DNS) for Let's Encrypt:

example.com. IN CAA 0 issue "letsencrypt.org"
External resources

🔰 Define security policies with security.txt

Rationale

The main purpose of security.txt is to help make things easier for companies and security researchers when trying to secure platforms. It also provides information to assist in disclosing security vulnerabilities.

When security researchers detect potential vulnerabilities in a page or application, they will try to contact someone "appropriate" to "responsibly" reveal the problem. It's worth taking care of getting to the right address.

This file should be placed under the /.well-known/ path, e.g. /.well-known/security.txt (RFC5785) of a domain name or IP address for web properties.

Example
curl -ks https://example.com/.well-known/security.txt

Contact: security@example.com
Contact: +1-209-123-0123
Encryption: https://example.com/pgp.txt
Preferred-Languages: en
Canonical: https://example.com/.well-known/security.txt
Policy: https://example.com/security-policy.html

And from Google:

curl -ks https://www.google.com/.well-known/security.txt

Contact: https://g.co/vulnz
Contact: mailto:security@google.com
Encryption: https://services.google.com/corporate/publickey.txt
Acknowledgements: https://bughunter.withgoogle.com/
Policy: https://g.co/vrp
Hiring: https://g.co/SecurityPrivacyEngJobs
# Flag: BountyCon{075e1e5eef2bc8d49bfe4a27cd17f0bf4b2b85cf}
External resources

Configuration Examples

Remember to make a copy of the current configuration and all files/directories.

Installation

I used step-by-step tutorial from Installation from source.

Configuration

Configuration of Google Cloud instance:

ITEM VALUE COMMENT
VM Google Cloud Platform
vCPU 2x
Memory 4096MB
HTTP Varnish on port 80
HTTPS NGINX on port 443

Reverse Proxy

This chapter describes the basic configuration of my proxy server (for blkcipher.info domain).

Configuration of my Reverse Proxy server is based on installation from source chapter. If you go through the installation process step by step you can use the following configuration (minor adjustments may be required).

Import configuration

It's very simple - clone the repo, backup your current configuration and perform full directory sync:

git clone https://github.com/trimstray/nginx-admins-handbook.git

tar czvfp ~/nginx.etc.tgz /etc/nginx && mv /etc/nginx /etc/nginx.old

rsync -avur lib/nginx/ /etc/nginx/

If you compiled NGINX you should also update/refresh modules. All compiled modules are stored in /usr/local/src/nginx-${ngx_version}/master/objs and installed in accordance with the value of the --modules-path variable.

Set bind IP address

Find and replace 192.168.252.2 string in directory and file names
cd /etc/nginx
find . -depth -name '*192.168.252.2*' -execdir bash -c 'mv -v "$1" "${1//192.168.252.2/xxx.xxx.xxx.xxx}"' _ {} \;
Find and replace 192.168.252.2 string in configuration files
cd /etc/nginx
find . -type f -print0 | xargs -0 sed -i 's/192.168.252.2/xxx.xxx.xxx.xxx/g'

Set your domain name

Find and replace blkcipher.info string in directory and file names
cd /etc/nginx
find . -depth -name '*blkcipher.info*' -execdir bash -c 'mv -v "$1" "${1//blkcipher.info/example.com}"' _ {} \;
Find and replace blkcipher.info string in configuration files
cd /etc/nginx
find . -type f -print0 | xargs -0 sed -i 's/blkcipher_info/example_com/g'
find . -type f -print0 | xargs -0 sed -i 's/blkcipher.info/example.com/g'

Regenerate private keys and certs

For localhost
cd /etc/nginx/master/_server/localhost/certs

# Private key + Self-signed certificate:
( _fd="localhost.key" ; _fd_crt="nginx_localhost_bundle.crt" ; \
openssl req -x509 -newkey rsa:2048 -keyout ${_fd} -out ${_fd_crt} -days 365 -nodes \
-subj "/C=X0/ST=localhost/L=localhost/O=localhost/OU=X00/CN=localhost" )
For default_server
cd /etc/nginx/master/_server/defaults/certs

# Private key + Self-signed certificate:
( _fd="defaults.key" ; _fd_crt="nginx_defaults_bundle.crt" ; \
openssl req -x509 -newkey rsa:2048 -keyout ${_fd} -out ${_fd_crt} -days 365 -nodes \
-subj "/C=X1/ST=default/L=default/O=default/OU=X11/CN=default_server" )
For your domain (e.g. Let's Encrypt)
cd /etc/nginx/master/_server/example.com/certs

# For multidomain:
certbot certonly -d example.com -d www.example.com --rsa-key-size 2048

# For wildcard:
certbot certonly --manual --preferred-challenges=dns -d example.com -d *.example.com --rsa-key-size 2048

# Copy private key and chain:
cp /etc/letsencrypt/live/example.com/fullchain.pem nginx_example.com_bundle.crt
cp /etc/letsencrypt/live/example.com/privkey.pem example.com.key

Add new domain

Updated nginx.conf
# At the end of the file (in 'IPS/DOMAINS' section):
include /etc/nginx/master/_server/domain.com/servers.conf;
include /etc/nginx/master/_server/domain.com/backends.conf;
Init domain directory
cd /etc/nginx/cd master/_server
cp -R example.com domain.com

cd domain.com
find . -depth -name '*example.com*' -execdir bash -c 'mv -v "$1" "${1//example.com/domain.com}"' _ {} \;
find . -type f -print0 | xargs -0 sed -i 's/example_com/domain_com/g'
find . -type f -print0 | xargs -0 sed -i 's/example.com/domain.com/g'

Create log directories

mkdir -p /var/log/nginx/localhost
mkdir -p /var/log/nginx/defaults
mkdir -p /var/log/nginx/others
mkdir -p /var/log/nginx/domains/blkcipher.info

chown -R nginx:nginx /var/log/nginx

Logrotate configuration

cp /etc/nginx/snippets/logrotate.d/nginx /etc/logrotate.d/

Test your configuration

nginx -t -c /etc/nginx/nginx.conf
You can’t perform that action at this time.