Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fluidasserts: init at 20.1.22554 #77787

Merged
merged 2 commits into from Jan 16, 2020
Merged

fluidasserts: init at 20.1.22554 #77787

merged 2 commits into from Jan 16, 2020

Conversation

@kamadorueda
Copy link
Member

kamadorueda commented Jan 15, 2020

Motivation for this change

It's a pretty nice framework that you can use to build 'exploits' (scripts) to verify if previous vulnerabilities found in your systems are still open or closed, the complexity of such scripts is up to the developer, and that's what make it so powerful

It can be used as CLI tool too, to run the exploits, or with predefined (basic) tests like --http or --aws

We use it actively every day and therefore we would like to get it directly from nixpkgs

It's open source https://fluidattacks.com/asserts/

Things done
  • Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
  • Built on platform(s)
    • NixOS
    • macOS
    • other Linux distributions
  • Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
  • Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
  • Tested execution of all binary files (usually in ./result/bin/)
  • Determined the impact on package closure size (by running nix path-info -S before and after)
  • Ensured that relevant documentation is up to date
  • Fits CONTRIBUTING.md.

nix-linter

$ nix-linter . && echo 'ok' || echo 'fail'
ok

nix-review

kamado:~/Documents/nixpkgs$ nix-review rev HEAD
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nix-review/0
From https://github.com/NixOS/nixpkgs
   f2251b483c0..3b8c2ac0000  master     -> refs/nix-review/0
$ git worktree add /home/kamado/.cache/nix-review/rev-fe32b55afa5923803957e425a9e0fdb7387e1d60/nixpkgs 3b8c2ac00004813e0478715b99caf60a79388314
Preparing worktree (detached HEAD 3b8c2ac0000)
Updating files: 100% (20761/20761), done.
HEAD is now at 3b8c2ac0000 Merge pull request #77776 from dtzWill/update/bibata-cursors-0.4.2
$ nix-env -f /home/kamado/.cache/nix-review/rev-fe32b55afa5923803957e425a9e0fdb7387e1d60/nixpkgs -qaP --xml --out-path --show-trace
$ git merge --no-commit fe32b55afa5923803957e425a9e0fdb7387e1d60
Updating 3b8c2ac0000..fe32b55afa5
Fast-forward
 pkgs/development/python-modules/fluidasserts/default.nix           | 142 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 pkgs/development/python-modules/fluidasserts/overlays/mixpanel.nix |  29 +++++++++++++
 pkgs/top-level/all-packages.nix                                    |   2 +
 pkgs/top-level/python-packages.nix                                 |   4 ++
 4 files changed, 177 insertions(+)
 create mode 100644 pkgs/development/python-modules/fluidasserts/default.nix
 create mode 100644 pkgs/development/python-modules/fluidasserts/overlays/mixpanel.nix
$ nix-env -f /home/kamado/.cache/nix-review/rev-fe32b55afa5923803957e425a9e0fdb7387e1d60/nixpkgs -qaP --xml --out-path --show-trace --meta
$ nix build --no-link --keep-going --max-jobs 4 --option build-use-sandbox true -f /home/kamado/.cache/nix-review/rev-fe32b55afa5923803957e425a9e0fdb7387e1d60/build.nix
[1 built, 0.0 MiB DL]
1 package were built:
fluidasserts

$ nix-shell /home/kamado/.cache/nix-review/rev-fe32b55afa5923803957e425a9e0fdb7387e1d60/shell.nix
these paths will be fetched (1.51 MiB download, 8.27 MiB unpacked):
  /nix/store/6lds7pywa1mvay194h6har2cdkhqw2ly-bash-interactive-4.4-p23
  /nix/store/d2iqwqxxq19jyvakh7l4jfvwnbz9rkka-bash-interactive-4.4-p23-dev
  /nix/store/g0wql6lf0ysg7kb6ymmarz6x8glds8if-bash-interactive-4.4-p23-info
  /nix/store/im6sdk69k5s7c8jxzhzfvb2j9nwg7hmg-bash-interactive-4.4-p23-man
  /nix/store/ppa5zzafkz86aaipg5lb3yvyih21p7jd-bash-interactive-4.4-p23-doc
  /nix/store/w95z96r4cb86hnl92xxrnx3m4mk6ixya-readline-7.0p5
copying path '/nix/store/ppa5zzafkz86aaipg5lb3yvyih21p7jd-bash-interactive-4.4-p23-doc' from 'https://cache.nixos.org'...
copying path '/nix/store/g0wql6lf0ysg7kb6ymmarz6x8glds8if-bash-interactive-4.4-p23-info' from 'https://cache.nixos.org'...
copying path '/nix/store/im6sdk69k5s7c8jxzhzfvb2j9nwg7hmg-bash-interactive-4.4-p23-man' from 'https://cache.nixos.org'...
copying path '/nix/store/w95z96r4cb86hnl92xxrnx3m4mk6ixya-readline-7.0p5' from 'https://cache.nixos.org'...
copying path '/nix/store/6lds7pywa1mvay194h6har2cdkhqw2ly-bash-interactive-4.4-p23' from 'https://cache.nixos.org'...
copying path '/nix/store/d2iqwqxxq19jyvakh7l4jfvwnbz9rkka-bash-interactive-4.4-p23-dev' from 'https://cache.nixos.org'...
innovation

[nix-shell:~/.cache/nix-review/rev-fe32b55afa5923803957e425a9e0fdb7387e1d60]$ asserts --help
usage: asserts [-h] [-q] [-k] [-n] [-o] [-c] [-u] [-ms] [-eec] [-mp] [-O FILE]
               [--http URL [URL ...]]
               [--ssl IP_ADDRESS:PORT [IP_ADDRESS:PORT ...]]
               [--dns NS [NS ...]] [--apk APK [APK ...]]
               [--lang FILE/DIR [FILE/DIR ...]]
               [--aws AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY [AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY ...]]
               [--azure AZURE_SUBSCRIPTION_ID:AZURE_CLIENT_ID:AZURE_CLIENT_SECRET:AZURE_TENANT_ID [AZURE_SUBSCRIPTION_ID:AZURE_CLIENT_ID:AZURE_CLIENT_SECRET:AZURE_TENANT_ID ...]]
               [--cloudformation FILE/DIR [FILE/DIR ...]]
               [exploits [exploits ...]]

positional arguments:
  exploits              exploits to execute

optional arguments:
  -h, --help            show this help message and exit
  -q, --quiet           do not show checks output
  -k, --kiss            keep it simple, shows only who and where has been
                        found to be vulnerable
  -n, --no-color        remove colors
  -o, --show-open       show only opened checks
  -c, --show-closed     show only closed checks
  -u, --show-unknown    show only unknown (error) checks
  -ms, --show-method-stats
                        show method-level stats at the end
  -eec, --enrich-exit-codes
                        make the exit codes more expressive
  -mp, --multiprocessing
                        enable multiprocessing over the provided list of
                        exploits.The number of used cpu cores defaults to the
                        local cpu count provided by the OS.
  -O FILE, --output FILE
                        save output in FILE
  --http URL [URL ...]  perform generic HTTP checks over given URL
  --ssl IP_ADDRESS:PORT [IP_ADDRESS:PORT ...]
                        perform generic SSL checks over given IP address and
                        port, if port is not specified it defaults to 443
  --dns NS [NS ...]     perform generic DNS checks over given nameserver
  --apk APK [APK ...]   perform generic APK checks over given APK file(s)
  --lang FILE/DIR [FILE/DIR ...]
                        perform static security checks over given files or
                        directories
  --aws AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY [AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY ...]
                        perform AWS checks using the given credentials
  --azure AZURE_SUBSCRIPTION_ID:AZURE_CLIENT_ID:AZURE_CLIENT_SECRET:AZURE_TENANT_ID [AZURE_SUBSCRIPTION_ID:AZURE_CLIENT_ID:AZURE_CLIENT_SECRET:AZURE_TENANT_ID ...]
                        perform Azure checks using the given credentials
  --cloudformation FILE/DIR [FILE/DIR ...]
                        perform AWS checks over CloudFormation templates
                        starting recursively from FILE/DIR

[nix-shell:~/.cache/nix-review/rev-fe32b55afa5923803957e425a9e0fdb7387e1d60]$ asserts --show-open --kiss --multiprocessing --http https://fluidattacks.com
#     ________      _     __   ___                        __
#    / ____/ /_  __(_)___/ /  /   |  _____________  _____/ /______
#   / /_  / / / / / / __  /  / /| | / ___/ ___/ _ \/ ___/ __/ ___/
#  / __/ / / /_/ / / /_/ /  / ___ |(__  |__  )  __/ /  / /_(__  )
# /_/   /_/\__,_/_/\__,_/  /_/  |_/____/____/\___/_/   \__/____/
#
# v. 20.1.21342
#  ___
# | >>|> fluid
# |___|  attacks, we hack your software
#
---
finding: Fluid Asserts - Protocols - HTTP part 1 Module
  check: fluidasserts.proto.http -> has_clear_viewstate
  check: fluidasserts.proto.http -> has_delete_method
  check: fluidasserts.proto.http -> has_dirlisting
  check: fluidasserts.proto.http -> has_host_header_injection
  check: fluidasserts.proto.http -> has_not_subresource_integrity
---
finding: Fluid Asserts - Protocols - HTTP part 2 Module
  check: fluidasserts.proto.http -> has_mixed_content
  check: fluidasserts.proto.http -> has_put_method
  check: fluidasserts.proto.http -> has_reverse_tabnabbing
  check: fluidasserts.proto.http -> has_sqli
  check: fluidasserts.proto.http -> has_trace_method
---
finding: Fluid Asserts - Protocols - HTTP part 3 Module
  check: fluidasserts.proto.http -> is_basic_auth_enabled
  check: fluidasserts.proto.http -> is_date_unsyncd
  check: fluidasserts.proto.http -> is_header_access_control_allow_origin_missing
  check: fluidasserts.proto.http -> is_header_cache_control_missing
  check: fluidasserts.proto.http -> is_header_content_security_policy_missing
  check: fluidasserts.proto.http -> is_header_content_type_missing
  check: fluidasserts.proto.http -> is_header_expires_missing
  check: fluidasserts.proto.http -> is_header_hsts_missing
  check: fluidasserts.proto.http -> is_header_perm_cross_dom_pol_missing
  check: fluidasserts.proto.http -> is_header_pragma_missing
---
finding: Fluid Asserts - Protocols - HTTP part 4 Module
  check: fluidasserts.proto.http -> is_header_server_present
  check: fluidasserts.proto.http -> is_header_x_asp_net_version_present
  check: fluidasserts.proto.http -> is_header_x_content_type_options_missing
  check: fluidasserts.proto.http -> is_header_x_frame_options_missing
  check: fluidasserts.proto.http -> is_header_x_powered_by_present
  check: fluidasserts.proto.http -> is_header_x_xxs_protection_missing
  check: fluidasserts.proto.http -> is_not_https_required
  check: fluidasserts.proto.http -> is_resource_accessible
  check: fluidasserts.proto.http -> is_response_delayed
  check: fluidasserts.proto.http -> is_sessionid_exposed
  check: fluidasserts.proto.http -> is_version_visible
---
finding: Fluid Asserts - Protocols - HTTP part 1 Module
---
check: fluidasserts.proto.http -> has_not_subresource_integrity
description: Check if elements fetched by the provided url have SRI.
status: OPEN
vulnerabilities:
- where: https://fluidattacks.com
  specific: link element has not integrity attributes
- where: https://fluidattacks.com
  specific: script element has not integrity attributes
risk: medium
---
finding: Fluid Asserts - Protocols - HTTP part 2 Module
---
check: fluidasserts.proto.http -> has_mixed_content
description: Check if resource has mixed (HTTP and HTTPS) links.
status: OPEN
vulnerabilities:
- where: https://fluidattacks.com/web/
  specific: Resource has mixed content
risk: low
---
finding: Fluid Asserts - Protocols - HTTP part 3 Module
---
check: fluidasserts.proto.http -> is_header_cache_control_missing
description: Check if Cache-Control HTTP header is properly set.
status: OPEN
vulnerabilities:
- where: https://fluidattacks.com/web/
  specific: Cache-Control header is missing which is insecure
risk: low
---
check: fluidasserts.proto.http -> is_header_content_security_policy_missing
description: Check if Content-Security-Policy HTTP header is properly set.
status: OPEN
vulnerabilities:
- where: https://fluidattacks.com/web/
  specific: Content-Security-Policy header is insecure
risk: medium
---
check: fluidasserts.proto.http -> is_header_content_type_missing
description: Check if Content-Type HTTP header is properly set.
status: OPEN
vulnerabilities:
- where: https://fluidattacks.com/web/
  specific: Content-Type header is insecure
risk: low
---
check: fluidasserts.proto.http -> is_header_hsts_missing
description: Check if Strict-Transport-Security HTTP header is properly set.
status: OPEN
vulnerabilities:
- where: https://fluidattacks.com/web/
  specific: Strict-Transport-Security is secure
risk: medium
---
finding: Fluid Asserts - Protocols - HTTP part 4 Module
---
check: fluidasserts.proto.http -> is_resource_accessible
description: Check if URL is available by checking response code.
status: OPEN
vulnerabilities:
- where: https://fluidattacks.com/web/
  specific: Resource is available
risk: medium
---
summary:
  test time: 5.8919 seconds
  checks:
    total: 31 (100%)
    errors: 0 (0.00%)
    unknown: 1 (3.23%)
    closed: 23 (74.19%)
    opened: 7 (22.58%)
  risk:
    high: 0 (0.00%)
    medium: 4 (57.14%)
    low: 3 (42.86%)
@kamadorueda

This comment has been minimized.

Copy link
Member Author

kamadorueda commented Jan 16, 2020

@jonringer

  • solved the default case on fetchPyPi
  • added many tests (didn't need to fetch via git, just shipped the tests to pypi)
  • upgraded mixpanel, (fortunately it wasn't being used by anyone else, so there is no danger in the upgrade)

thanks for the review, we made some nice changes to the package thanks to it

nixpkgs$ nix-review rev HEAD
$ git -c fetch.prune=false fetch --force https://github.com/NixOS/nixpkgs master:refs/nix-review/0
From https://github.com/NixOS/nixpkgs
   33941e49f2f..5a003383fba  master     -> refs/nix-review/0
$ git worktree add /home/kamado/.cache/nix-review/rev-80d6f0dd41bab75968b08943f1c5e64f108f8f20/nixpkgs 5a003383fbadef7278c3c1a8ec454b9a2f6d8e51
Preparing worktree (detached HEAD 5a003383fba)
Updating files: 100% (20767/20767), done.
HEAD is now at 5a003383fba pythonPackages.pyTelegramBotAPI: Init at 3.6.6 (#65509)
$ nix-env -f /home/kamado/.cache/nix-review/rev-80d6f0dd41bab75968b08943f1c5e64f108f8f20/nixpkgs -qaP --xml --out-path --show-trace
$ git merge --no-commit 80d6f0dd41bab75968b08943f1c5e64f108f8f20
Updating 5a003383fba..80d6f0dd41b
Fast-forward
 pkgs/development/python-modules/fluidasserts/default.nix | 188 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 pkgs/development/python-modules/mixpanel/default.nix     |  50 +++++++++++++-------
 pkgs/top-level/all-packages.nix                          |   2 +
 pkgs/top-level/python-packages.nix                       |   2 +
 4 files changed, 225 insertions(+), 17 deletions(-)
 create mode 100644 pkgs/development/python-modules/fluidasserts/default.nix
$ nix-env -f /home/kamado/.cache/nix-review/rev-80d6f0dd41bab75968b08943f1c5e64f108f8f20/nixpkgs -qaP --xml --out-path --show-trace --meta
$ nix build --no-link --keep-going --max-jobs 4 --option build-use-sandbox true -f /home/kamado/.cache/nix-review/rev-80d6f0dd41bab75968b08943f1c5e64f108f8f20/build.nix
[2 built, 0.0 MiB DL]
2 package were built:
fluidasserts python37Packages.mixpanel
@kamadorueda kamadorueda force-pushed the kamadorueda:fluidasserts branch from fe32b55 to 80d6f0d Jan 16, 2020
kamadorueda added 2 commits Jan 15, 2020
@kamadorueda kamadorueda force-pushed the kamadorueda:fluidasserts branch from 80d6f0d to 4512861 Jan 16, 2020
@kamadorueda kamadorueda changed the title fluidasserts: init at 20.1.21324 fluidasserts: init at 20.1.22554 Jan 16, 2020
@ofborg ofborg bot added the 8.has: clean-up label Jan 16, 2020
Copy link
Contributor

jonringer left a comment

diff LGTM
commits LGTM

[5 built, 65 copied (276.3 MiB), 30.5 MiB DL]
https://github.com/NixOS/nixpkgs/pull/77787
2 package built:
fluidasserts python37Packages.mixpanel
test/test_cloud_aws_cloudformation_rds.py \
test/test_cloud_aws_cloudformation_s3.py \
test/test_cloud_aws_cloudformation_secretsmanager.py \
test/test_format_apk.py \

This comment has been minimized.

Copy link
@jonringer

jonringer Jan 16, 2020

Contributor

you could use bash expansion to make this not so repetitive:

Suggested change
test/test_format_apk.py \
test/test_format_{apk,file,jks,jwt,pdf,pkcs12,string}.py \

example:

$ echo test/test_format_{apk,file,jks,jwt,pdf,pkcs12,string}.py
test/test_format_apk.py test/test_format_file.py test/test_format_jks.py test/test_format_jwt.py test/test_format_pdf.py test/test_format_pkcs12.py test/test_format_string.py
@jonringer

This comment has been minimized.

Copy link
Contributor

jonringer commented Jan 16, 2020

@GrahamcOfBorg build fluidasserts python37Packages.mixpanel

@jonringer

This comment has been minimized.

Copy link
Contributor

jonringer commented Jan 16, 2020

darwin failure unrelated

@jonringer jonringer merged commit 7d3e3cd into NixOS:master Jan 16, 2020
18 checks passed
18 checks passed
fluidasserts, python37Packages.mixpanel on x86_64-darwin Failure
Details
Evaluation Performance Report Evaluator Performance Report
Details
fluidasserts on aarch64-linux Success
Details
fluidasserts on x86_64-linux Success
Details
fluidasserts, python37Packages.mixpanel on aarch64-linux Success
Details
fluidasserts, python37Packages.mixpanel on x86_64-linux Success
Details
grahamcofborg-eval ^.^!
Details
grahamcofborg-eval-check-maintainers matching changed paths to changed attrs...
Details
grahamcofborg-eval-check-meta config.nix: checkMeta = true
Details
grahamcofborg-eval-darwin nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A darwin-tested
Details
grahamcofborg-eval-nixos nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release-combined.nix -A tested
Details
grahamcofborg-eval-nixos-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A manual
Details
grahamcofborg-eval-nixos-options nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./nixos/release.nix -A options
Details
grahamcofborg-eval-nixpkgs-manual nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A manual
Details
grahamcofborg-eval-nixpkgs-tarball nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A tarball
Details
grahamcofborg-eval-nixpkgs-unstable-jobset nix-instantiate --arg nixpkgs { outPath=./.; revCount=999999; shortRev="ofborg"; } ./pkgs/top-level/release.nix -A unstable
Details
grahamcofborg-eval-package-list nix-env -qa --json --file .
Details
grahamcofborg-eval-package-list-no-aliases nix-env -qa --json --file . --arg config { allowAliases = false; }
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

2 participants
You can’t perform that action at this time.