Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

aws signer: fix missing x-content-sha256 header #3601

Merged
merged 8 commits into from
May 1, 2023

Conversation

tarunKoyalwar
Copy link
Member

@tarunKoyalwar tarunKoyalwar commented Apr 24, 2023

Proposed changes

Checklist

  • Pull request is created against the dev branch
  • All checks passed (lint, unit/integration/regression tests etc.) with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

@tarunKoyalwar tarunKoyalwar self-assigned this Apr 24, 2023
@tarunKoyalwar tarunKoyalwar linked an issue Apr 24, 2023 that may be closed by this pull request
@tarunKoyalwar tarunKoyalwar requested review from princechaddha and removed request for princechaddha April 24, 2023 13:51
@tarunKoyalwar tarunKoyalwar marked this pull request as draft April 24, 2023 13:55
@tarunKoyalwar
Copy link
Member Author

tarunKoyalwar commented Apr 26, 2023

POC

id: create-s3-bucket
info:
  name: Create S3 Bucket
  author: princechaddha
  severity: info
  description: |
    Creadted s3 bucket in you amazon account
  reference:
    - https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeVpcs.html
    - https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeFlowLogs.html
  remediation: Enabling VPC Flow Logs
  tags: cloud,devops,aws,amazon,cloudtrail,s3

self-contained: true
signature: aws
variables:
  service: s3
  region: us-east-1
  bucketname: demodaytest6789

requests:
  - raw:
      - |
        PUT https://{{bucketname}}.s3.{{region}}.amazonaws.com/ HTTP/1.1
        User-Agent: aws-cli/2.9.15 Python/3.11.1 Darwin/22.3.0 source/arm64 prompt/off command/s3api.create-bucket

    iterate-all: true
    matchers-condition: and
    matchers:
      - type: word
        part: body
        words:
          - '</LoggingEnabled>'
$ go run . -t ~/test-templates/s3bucket.yaml -debug -V "bucketname=osstest123"

                     __     _
   ____  __  _______/ /__  (_)
  / __ \/ / / / ___/ / _ \/ /
 / / / / /_/ / /__/ /  __/ /
/_/ /_/\__,_/\___/_/\___/_/   v2.9.3-dev

		projectdiscovery.io

[INF] Current nuclei version: v2.9.3-dev (development)
[INF] Current nuclei-templates version: v9.4.3 (latest)
[INF] New templates added in latest release: 55
[INF] Templates loaded for current scan: 1
[INF] [create-s3-bucket] Dumped HTTP request for https://osstest123.s3.us-east-1.amazonaws.com/

PUT / HTTP/1.1
Host: osstest123.s3.us-east-1.amazonaws.com
User-Agent: aws-cli/2.9.15 Python/3.11.1 Darwin/22.3.0 source/arm64 prompt/off command/s3api.create-bucket
Connection: close
Content-Length: 0
Authorization: xyz
X-Amz-Content-Sha256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
X-Amz-Date: 20230426T191357Z
Accept-Encoding: gzip

[DBG] [create-s3-bucket] Dumped HTTP response https://osstest123.s3.us-east-1.amazonaws.com/

HTTP/1.1 200 OK
Connection: close
Content-Length: 0
Date: Wed, 26 Apr 2023 19:13:59 GMT
Location: /osstest123
Server: AmazonS3
X-Amz-Id-2: 8VAu9zXrQoVKXNepr0Ekl3L1whW5Goub8rqGLM2croEazL0i5V3SeduYd9R3H/hcYxubxWpdP7k=
X-Amz-Request-Id: 081BFG0XX8M19NCT

[INF] No results found. Better luck next time!

@tarunKoyalwar tarunKoyalwar marked this pull request as ready for review April 26, 2023 19:25
@tarunKoyalwar tarunKoyalwar marked this pull request as draft April 26, 2023 20:04
@tarunKoyalwar
Copy link
Member Author

Race request bug fix

Before

$  time ./nuclei -u scanme.sh -t ../../../integration_tests/http/race-simple.yaml -duc -ts -silent  
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:17:51] [race-condition-testing] [http] [info] https://scanme.sh/
./nuclei -u scanme.sh -t ../../../integration_tests/http/race-simple.yaml -du  0.13s user 0.08s system 0% cpu 23.371 total

After

$ time ./nuclei -u scanme.sh -t ../../../integration_tests/http/race-simple.yaml -duc -ts -silent
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
[2023-04-27 02:19:12] [race-condition-testing] [http] [info] https://scanme.sh/
./nuclei -u scanme.sh -t ../../../integration_tests/http/race-simple.yaml -du  0.14s user 0.09s system 16% cpu 1.402 total

@tarunKoyalwar tarunKoyalwar marked this pull request as ready for review April 26, 2023 21:22
@tarunKoyalwar tarunKoyalwar added the Status: Review Needed The issue has a PR attached to it which needs to be reviewed label Apr 27, 2023
@tarunKoyalwar tarunKoyalwar requested review from ShubhamRasal and removed request for Mzack9999 April 28, 2023 14:56
Copy link
Contributor

@ShubhamRasal ShubhamRasal left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@ehsandeep ehsandeep merged commit 7f5e4e2 into dev May 1, 2023
11 checks passed
@ehsandeep ehsandeep deleted the issue-3597-content-sha256-header branch May 1, 2023 06:45
@ehsandeep ehsandeep removed the Status: Review Needed The issue has a PR attached to it which needs to be reviewed label May 1, 2023
RamanaReddy0M pushed a commit that referenced this pull request May 3, 2023
* fix missing x-content-sha256 header

* fix variable priority in self-contained templates

* remove debug statement

* adds generic raw request parser for self-contained req

* more integration tests

* bug fix: 10x faster race requests

* fix failing integration test
ehsandeep added a commit that referenced this pull request May 9, 2023
* Add utility to write max-requests to templates

* fix lint error

* fix max-request update edge case

* fix convert max-request: 1 => max-request: 1

* WIP, most of the code is commented

* Refactor the find and replace  logic

* Skip if template has the max-requests, do not overwrite

- return errors
- add warnings

* Fix the wrong index calculation

- Refactor the getInfoBlock function to not compile regex everytime

* Update -tc flag to filter fields within the classification section (#3606)

* Add fields from Classification section in a template to the -tc flag expression evaluation

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

* Add tests for filtering Classification section using -tc flag

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

* Fix hyphenated Metadata keys beings added to parameters

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

* Add tests to the fix for hyphenated fields encountered in Metadata section

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

---------

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

* Check severity att while validating (#3540)

* Make severity attribute required

* Update test err msg

* minor

* Do not strict check serverity

* Fix failing test

* Don't print warning in workflow loader

- workflow loader that contains tags load all the template and parse it
- i.e it iw printing warning recursively, ignore as the templates
  already getting valiated

* Fix error typo

* Resolve comments

- split the function into two diff

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>

* tlsx dep update (#3620)

* updated interactsh version (#3621)

* updated interactsh version

* workflow update

* aws signer: fix missing x-content-sha256 header (#3601)

* fix missing x-content-sha256 header

* fix variable priority in self-contained templates

* remove debug statement

* adds generic raw request parser for self-contained req

* more integration tests

* bug fix: 10x faster race requests

* fix failing integration test

* chore(deps): bump github.com/xanzy/go-gitlab in /v2 (#3624)

Bumps [github.com/xanzy/go-gitlab](https://github.com/xanzy/go-gitlab) from 0.82.0 to 0.83.0.
- [Release notes](https://github.com/xanzy/go-gitlab/releases)
- [Changelog](https://github.com/xanzy/go-gitlab/blob/master/releases_test.go)
- [Commits](xanzy/go-gitlab@v0.82.0...v0.83.0)

---
updated-dependencies:
- dependency-name: github.com/xanzy/go-gitlab
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/miekg/dns from 1.1.53 to 1.1.54 in /v2 (#3625)

Bumps [github.com/miekg/dns](https://github.com/miekg/dns) from 1.1.53 to 1.1.54.
- [Release notes](https://github.com/miekg/dns/releases)
- [Changelog](https://github.com/miekg/dns/blob/master/Makefile.release)
- [Commits](miekg/dns@v1.1.53...v1.1.54)

---
updated-dependencies:
- dependency-name: github.com/miekg/dns
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* chore(deps): bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /v2 (#3626)

Bumps [github.com/aws/aws-sdk-go-v2/feature/s3/manager](https://github.com/aws/aws-sdk-go-v2) from 1.11.61 to 1.11.64.
- [Release notes](https://github.com/aws/aws-sdk-go-v2/releases)
- [Changelog](https://github.com/aws/aws-sdk-go-v2/blob/main/CHANGELOG.md)
- [Commits](aws/aws-sdk-go-v2@feature/s3/manager/v1.11.61...feature/s3/manager/v1.11.64)

---
updated-dependencies:
- dependency-name: github.com/aws/aws-sdk-go-v2/feature/s3/manager
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Fix check for OS made in MustDisableSandbox() (#3631)

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

* Fix wrong template loading in dev branch (#3629)

* Templates wrong loading

* Add tests to cover following scenarios

- check optional fields only if template loaded
- it should return warning only if template is loaded

* enable color in windows (#3634)

* enable color in windows

* fixed win workflow

* typo update

* tlsx dep update (#3633)

* tlsx dep update

* upgrde httpx => 1.3.0

* Fix check for OS made in MustDisableSandbox() (#3631)

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>

* Fix wrong template loading in dev branch (#3629)

* Templates wrong loading

* Add tests to cover following scenarios

- check optional fields only if template loaded
- it should return warning only if template is loaded

* enable color in windows (#3634)

* enable color in windows

* fixed win workflow

* typo update

---------

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>
Co-authored-by: Ramana Reddy <ramanaredy.manda@gmail.com>
Co-authored-by: Suraj Kamath <kamathsuraj95@gmail.com>
Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>

* Expose DNS fields for matchers and extractors (#3613)

* Extend dns extractor to dns answer records

* add test template

* Ignore error for dns variables are not found

* Add all the records of answer section

* Fixed the wrong typecasting

* Issue 3564 var override (#3599)

* Check if the variables are override by other means

- you can override the template variable value using command line flags

* Update lazy eval logic

- previously, we were checking any function/expression in variable
- now, update the logic, lazy eval only if variable contains any
  protocol variable(global)

* add integration tests

* Add test to check the dsl function working in variable

* gather all generate variables logic in utils

* go mod update

* Refactor the generate variables function

* go mod update+ fix typo

---------

Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>

* update rod to v0.112.9 #3552 (#3637)

* update rod to v0.112.9

* removed unused reflection

---------

Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>

* change max-requests label to max-request

---------

Signed-off-by: iamargus95 <kamathsuraj95@gmail.com>
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: shubhamrasal <shubhamdharmarasal@gmail.com>
Co-authored-by: Suraj Kamath <kamathsuraj95@gmail.com>
Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com>
Co-authored-by: Mzack9999 <mzack9999@protonmail.com>
Co-authored-by: Sandeep Singh <sandeep@projectdiscovery.io>
Co-authored-by: Tarun Koyalwar <45962551+tarunKoyalwar@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Shubham Rasal <shubham@projectdiscovery.io>
Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
Co-authored-by: Tarun Koyalwar <tarun@projectdiscovery.io>
Co-authored-by: lu4nx <lx@shellcodes.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

X-Amz-Content-SHA256 header missing when using AWS signature
4 participants