Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Signing Key Expire within a day #70

Open
qdchong opened this issue Aug 23, 2019 · 20 comments
Open

Signing Key Expire within a day #70

qdchong opened this issue Aug 23, 2019 · 20 comments

Comments

@qdchong
Copy link

qdchong commented Aug 23, 2019

i encountered the error
AuthorizationHeaderMalformedThe authorization header is malformed; Invalid credential date. Date is not the same as X-Amz-Date.... after a day.

I thought the signing key was suppose to last for a week. can anyone help? How do i set the X-Amz-Date? Or do i need to set the amz-expire field??

@CpuID
Copy link

CpuID commented Aug 28, 2019

I've got the same issue, haven't resolved it yet.

@CpuID
Copy link

CpuID commented Sep 3, 2019

@qdchong to compare our implementations:

  • we generate a new signing key once a day right now ( related )
  • I've found that sometimes even after a SIGHUP with a new key in the configuration with a newly generated signing key, I still get AuthorizationHeaderMalformed errors which is really weird...

@CpuID
Copy link

CpuID commented Sep 5, 2019

@qdchong can you try implement something looking like https://tenzer.dk/nginx-with-dynamic-upstreams/ in your configs and see if it yields a difference for you? I did it about 2 days ago, so far rolled over UTC twice and haven't had an issue. Not sure if this is what did the job or not, but potentially...

@qdchong
Copy link
Author

qdchong commented Sep 5, 2019

Hmm wow i shall try it. I was looking at what other options r there. Thanks so much

@CpuID
Copy link

CpuID commented Nov 27, 2019

I ended up getting the The authorization header is malformed; Invalid credential date. Date is not the same as X-Amz-Date. again unfortunately... :( even with our approach in place. Haven't looked deeper at why yet.

@AdrianVandierAst
Copy link

Hello :)

The README of this module is wrong.
Look at https://docs.aws.amazon.com/general/latest/gr/sigv4-date-handling.html
The signing key IS usable only for requests signed the same date, maybe your problem is when request are created just before midnight and send just after.
I lost two days on that -_-

So, even if "It is an insecure practise to let the secret key reside on your nginx server.", we are all forced to have a cronned script at midnight exactly that regenerates the nginx configuration.

In fact, the module SHOULD generates the signing key since the promise of security is not possible.

I'm a bit disapointed by the date validation in the signature v4 protocol :(

@BrutalSimplicity
Copy link

BrutalSimplicity commented Jan 30, 2020

@AdrianVandierAst

I've got a proposed solution to this that uses temporary credentials and a python script that will regenerate a new signature upon expiration. It's limited by how often it gets run by cron (minutely), but you might be able to have it run continuously and check for expiration ahead of time. I've also thought about using lua for something a bit more robust that doesn't require a hot reload.

See the PR #74

@hexmarkrecords
Copy link

I wonder if we convert the python code to C++ and include it in the module. It could update the config struct in a lazy fashion. The date found in the key scope should be checked against the current date and if different update the struct with new key scope and signing key. The single requests around midnight UTC will run a little slower but subsequent requests will have normal performance.

@hexmarkrecords
Copy link

I'm currently working on a solution - my C is pretty rusty - I should really write some unit tests for it. Will post back once ive tested it in my pre-prod environment.

@hexmarkrecords
Copy link

hexmarkrecords commented Jun 4, 2020

I have produced a forked version which calculates the signing key when the date changes. It subsequently caches the signing key and key scope in the configuration. It does break the best practices of keeping the secret key off the nginx server.

I have simplified the nginx configuration to that similar to AWS2 config found in earlier versions. It still uses AWS4 for the signing method so no change there.

Not sure how you would do it otherwise as at 23:59:59:999 a request could be in flight and reach Amazon the following day at 00:00:00:000. Amazon would then deem the request erroneous by AWS as there would be a discrepancy between the header date, the signing key and key scope.

I have added some unit tests to test the new functionality.

By all means take from it what you wish. I agree with the original author intent of keeping the keys off the box, but couldn't see a way round it without over complicating the hack.

Fork here >> https://github.com/hexmarkrecords/ngx_aws_auth

If you would like to provide a better solution, poke holes, please chime in.

@hexmarkrecords
Copy link

I have tested it the following day and my assets still load:

https://hexmarkrecords.com

All dynamic assets are now served from S3 without expiry .... Yay

@leffuy
Copy link

leffuy commented Jun 18, 2020

@hexmarkrecords Trying to use your fork, as I couldn't get the original here to work at all.

Says "unknown direct aws_service" when using your example nginx.conf. Tried taking this out and get returned a 403 with "The request signature we calculated does not match the signature you provided"

using following code

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        #root         /usr/share/nginx/html;


        aws_access_key REDACTED; # Example AKIDEXAMPLE
        aws_secret_key REDACTED; #Example L4vRLWAO92X5L3Sqk5QydUSdB0nC9+1wfqLMOKLbRp4=
        aws_region us-west-2;
        # aws_service s3;
        aws_s3_bucket my.redacted.bucket;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
            aws_sign;
            # aws_endpoint "s3.us-west-2.amazonaws.com";
            #proxy_pass_request_headers off;
            proxy_pass https://my.redacted.bucket.s3.us-west-2.amazonaws.com/;
         }

    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers PROFILE=SYSTEM;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

@hexmarkrecords
Copy link

hexmarkrecords commented Jun 18, 2020

@hexmarkrecords Trying to use your fork, as I couldn't get the original here to work at all.

Says "unknown direct aws_service" when using your example nginx.conf. Tried taking this out and get returned a 403 with "The request signature we calculated does not match the signature you provided"

using following code

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;
    tcp_nopush          on;
    tcp_nodelay         on;
    keepalive_timeout   65;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  _;
        #root         /usr/share/nginx/html;


        aws_access_key REDACTED; # Example AKIDEXAMPLE
        aws_secret_key REDACTED; #Example L4vRLWAO92X5L3Sqk5QydUSdB0nC9+1wfqLMOKLbRp4=
        aws_region us-west-2;
        # aws_service s3;
        aws_s3_bucket my.redacted.bucket;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
            aws_sign;
            # aws_endpoint "s3.us-west-2.amazonaws.com";
            #proxy_pass_request_headers off;
            proxy_pass https://my.redacted.bucket.s3.us-west-2.amazonaws.com/;
         }

    }

# Settings for a TLS enabled server.
#
#    server {
#        listen       443 ssl http2 default_server;
#        listen       [::]:443 ssl http2 default_server;
#        server_name  _;
#        root         /usr/share/nginx/html;
#
#        ssl_certificate "/etc/pki/nginx/server.crt";
#        ssl_certificate_key "/etc/pki/nginx/private/server.key";
#        ssl_session_cache shared:SSL:1m;
#        ssl_session_timeout  10m;
#        ssl_ciphers PROFILE=SYSTEM;
#        ssl_prefer_server_ciphers on;
#
#        # Load configuration files for the default server block.
#        include /etc/nginx/default.d/*.conf;
#
#        location / {
#        }
#
#        error_page 404 /404.html;
#            location = /40x.html {
#        }
#
#        error_page 500 502 503 504 /50x.html;
#            location = /50x.html {
#        }
#    }

}

I have something like this:

http {
    server {
        listen     8001;

        aws_access_key      %AWS_ACCESS_KEY%;
        aws_secret_key      %AWS_SECRET_ACCESS_KEY%;
        aws_region          %AWS_REGION%;
        aws_s3_bucket       %BUCKET_NAME%;
        resolver 8.8.8.8;

        location / {
            access_log off;
            error_log off;

            aws_sign;
            proxy_pass http://%BUCKET_NAME%.s3.amazonaws.com;
        }
    }

And that worked for me bud. Ive just copied it from my config that is in production. Hopefully that should help you.

@hexmarkrecords
Copy link

i'm gonna look in to the aws_service param as that should be work also. It defaults to 's3' anyway.

@hexmarkrecords
Copy link

hexmarkrecords commented Jun 18, 2020

Spotted an issue with the service parameter so i will fix that - but in the mean time it defaults to 's3'. Thanks for letting me know @leffuy

@hexmarkrecords
Copy link

pushed to master and created new tag 2.2.2 - fixed missing service optional parameter

@mwangaben
Copy link

Hello @hexmarkrecords i am trying to compile this with nginx-1.19.0 with --add-module=/home/admin/building/ngx_aws_auth-2.2.2, but when i run make this is what i get

/home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c: In function ‘ngx_aws_auth__sign_hmac’: /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: storage size of ‘hmac’ isn’t known HMAC_CTX hmac; ^~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:59:5: error: implicit declaration of function ‘HMAC_CTX_init’ [-Werror=implicit-function-declaration] HMAC_CTX_init(&hmac); ^~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:60:5: error: ‘HMAC_Init’ is deprecated [-Werror=deprecated-declarations] HMAC_Init(&hmac, key, key_length, EVP_sha256()); ^~~~~~~~~ In file included from /usr/include/openssl/e_os2.h:13:0, from /usr/include/openssl/bio.h:13, from /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:10: /usr/include/openssl/hmac.h:28:1: note: declared here DEPRECATEDIN_1_1_0(__owur int HMAC_Init(HMAC_CTX *ctx, const void *key, int len, ^ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:63:5: error: implicit declaration of function ‘HMAC_CTX_cleanup’ [-Werror=implicit-function-declaration] HMAC_CTX_cleanup(&hmac); ^~~~~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: unused variable ‘hmac’ [-Werror=unused-variable] HMAC_CTX hmac; ^~~~ cc1: all warnings being treated as errors objs/Makefile:1317: recipe for target 'objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o' failed make[1]: *** [objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o] Error 1

Kindly help

@hexmarkrecords
Copy link

Hello @hexmarkrecords i am trying to compile this with nginx-1.19.0 with --add-module=/home/admin/building/ngx_aws_auth-2.2.2, but when i run make this is what i get

/home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c: In function ‘ngx_aws_auth__sign_hmac’: /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: storage size of ‘hmac’ isn’t known HMAC_CTX hmac; ^~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:59:5: error: implicit declaration of function ‘HMAC_CTX_init’ [-Werror=implicit-function-declaration] HMAC_CTX_init(&hmac); ^~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:60:5: error: ‘HMAC_Init’ is deprecated [-Werror=deprecated-declarations] HMAC_Init(&hmac, key, key_length, EVP_sha256()); ^~~~~~~~~ In file included from /usr/include/openssl/e_os2.h:13:0, from /usr/include/openssl/bio.h:13, from /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:10: /usr/include/openssl/hmac.h:28:1: note: declared here DEPRECATEDIN_1_1_0(__owur int HMAC_Init(HMAC_CTX *ctx, const void *key, int len, ^ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:63:5: error: implicit declaration of function ‘HMAC_CTX_cleanup’ [-Werror=implicit-function-declaration] HMAC_CTX_cleanup(&hmac); ^~~~~~~~~~~~~~~~ /home/admin/building/ngx_aws_auth-2.2.2/crypto_helper_openssl.c:58:14: error: unused variable ‘hmac’ [-Werror=unused-variable] HMAC_CTX hmac; ^~~~ cc1: all warnings being treated as errors objs/Makefile:1317: recipe for target 'objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o' failed make[1]: *** [objs/addon/ngx_aws_auth-2.2.2/crypto_helper_openssl.o] Error 1

Kindly help

I can take a look over the weekend.

@hexmarkrecords
Copy link

hexmarkrecords commented Aug 8, 2020

Hi @mwangaben I have now updated the branch (2.3.0 and master) to build a dynamic module exporting it to /private/tmp on the host machine when using docker.

It now works with Open SSL 1.1.1 and therefore compatible with Nginx 1.19.1.

Hope it helps you out.

@hexmarkrecords
Copy link

I wonder if I should try to turn this into a proper pull request for the main repos instead of on a fork?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants