Fedora requires the following prereqs (other distros may have similar requirements)
dnf install pcre-devel gd gd-devel GeoIP GeoIP-devel
Run
cd <your/path/to/repo>
./buildnginx.sh
Once complete, your compiled rpm will be in ../rpmbuilds
from where you ran buildnginx.sh
above.
Update nginx Version in the SPECS/nginx.spec file. You can use any version, including mainline, though we recommend you use the latest found here: http://nginx.org/en/download.html