Skip to content

v1.20.0: Thancred Waters

Latest
Latest
Compare
Choose a tag to compare
@Xe Xe released this 27 Jun 16:19
· 71 commits to main since this release
v1.20.0
c2423d0

Anubis now has support for weighing the soul of incoming requests with custom rules and thresholds. Anubis also can function without the use of client-side JavaScript using the metarefresh challenge.

The big ticket items are as follows:

  • Implement a no-JS challenge method: metarefresh (#95)
  • Implement request "weight", allowing administrators to customize the behaviour of Anubis based on specific criteria
  • Implement GeoIP and ASN based checks via Thoth (#206)
  • Add custom weight thresholds via CEL (#688)
  • Move Open Graph configuration to the policy file
  • Enable support for Open Graph metadata to be returned by default instead of doing lookups against the target
  • Add robots2policy CLI utility to convert robots.txt files to Anubis challenge policies using CEL expressions (#409)
  • Refactor challenge presentation logic to use a challenge registry
  • Allow challenge implementations to register HTTP routes
  • Imprint/Impressum support (#362)
  • Fix "invalid response" after "Success!" in Chromium (#564)

A lot of performance improvements have been made:

  • Replace internal SHA256 hashing with xxhash for 4-6x performance improvement in policy evaluation and cache operations
  • Optimized the OGTags subsystem with reduced allocations and runtime per request by up to 66%
  • Replace cidranger with bart for IP range checking, improving IP matching performance by 3-20x with zero heap
    allocations

And some cleanups/refactors were added:

  • Fix OpenGraph passthrough (#717)
  • Remove the unused /test-error endpoint and update the testing endpoint /make-challenge to only be enabled in
    development
  • Add --xff-strip-private flag/envvar to toggle skipping X-Forwarded-For private addresses or not
  • Bump AI-robots.txt to version 1.37
  • Make progress bar styling more compatible (UXP, etc)
  • Add --strip-base-prefix flag/envvar to strip the base prefix from request paths when forwarding to target servers
  • Fix an off-by-one in the default threshold config
  • Add functionality for HS512 JWT algorithm
  • Add support for dynamic cookie domains with the --cookie-dynamic-domain/COOKIE_DYNAMIC_DOMAIN flag/envvar

Request weight is one of the biggest ticket features in Anubis. This enables Anubis to be much closer to a Web Application Firewall and when combined with custom thresholds allows administrators to have Anubis take advanced reactions. For more information about request weight, see the request weight section of the policy file documentation.

TL;DR when you have one or more WEIGHT rules like this:

bots:
  - name: gitea-session-token
    action: WEIGH
    expression:
      all:
        - '"Cookie" in headers'
        - headers["Cookie"].contains("i_love_gitea=")
    # Remove 5 weight points
    weight:
      adjust: -5

You can configure custom thresholds like this:

thresholds:
  - name: minimal-suspicion # This client is likely fine, its soul is lighter than a feather
    expression: weight < 0 # a feather weighs zero units
    action: ALLOW # Allow the traffic through

  # For clients that had some weight reduced through custom rules, give them a
  # lightweight challenge.
  - name: mild-suspicion
    expression:
      all:
        - weight >= 0
        - weight < 10
    action: CHALLENGE
    challenge:
      # https://anubis.techaro.lol/docs/admin/configuration/challenges/metarefresh
      algorithm: metarefresh
      difficulty: 1
      report_as: 1

  # For clients that are browser-like but have either gained points from custom
  # rules or report as a standard browser.
  - name: moderate-suspicion
    expression:
      all:
        - weight >= 10
        - weight < 20
    action: CHALLENGE
    challenge:
      # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
      algorithm: fast
      difficulty: 2 # two leading zeros, very fast for most clients
      report_as: 2

  # For clients that are browser like and have gained many points from custom
  # rules
  - name: extreme-suspicion
    expression: weight >= 20
    action: CHALLENGE
    challenge:
      # https://anubis.techaro.lol/docs/admin/configuration/challenges/proof-of-work
      algorithm: fast
      difficulty: 4
      report_as: 4

These thresholds apply when no other ALLOW, DENY, or CHALLENGE rule matches the request. WEIGHT rules add and remove request weight as needed:

bots:
  - name: gitea-session-token
    action: WEIGH
    expression:
      all:
        - '"Cookie" in headers'
        - headers["Cookie"].contains("i_love_gitea=")
    # Remove 5 weight points
    weight:
      adjust: -5

  - name: bot-like-user-agent
    action: WEIGH
    expression: '"Bot" in userAgent'
    # Add 5 weight points
    weight:
      adjust: 5

Of note: the default "generic browser" rule assigns 10 weight points:

# Generic catchall rule
- name: generic-browser
  user_agent_regex: >-
    Mozilla|Opera
  action: WEIGH
  weight:
    adjust: 10

Adjust this as you see fit.

What's Changed

  • build(deps-dev): bump esbuild from 0.25.4 to 0.25.5 in the npm group by @dependabot in #600
  • build(deps): bump docker/build-push-action from 6.17.0 to 6.18.0 in the github-actions group by @dependabot in #602
  • build(deps): bump github.com/a-h/templ from 0.3.865 to 0.3.887 in the gomod group by @dependabot in #601
  • docs(faq): anubis does not mine bitcoin by @Xe in #609
  • feat: implement challenge registry by @Xe in #607
  • docs(known-instances): add Alliance of Hessian Libraries by @CryptoCopter in #611
  • docs(subrequest-auth): document required policy changes by @foosinn in #613
  • docs: Adjust the name of the cookie to the current "techaro.lol-anubis-auth" by @jieter in #615
  • fix(lib/challenge): allow challenges to register HTTP routes by @Xe in #620
  • docs(known-instances): add wiki.dolphin-emu.org to known instances by @lotharsm in #626
  • feat(lib/challenge): HTTP meta refresh challenge method by @Xe in #623
  • style: Some minor fixes by @JasonLovesDoggo in #548
  • Bump ai.robots.txt to v1.34 by @Dryusdan in #632
  • build(deps): bump the gomod group with 2 updates by @dependabot in #634
  • docs(admin/environments): Prefer IPv6 over IPv4 for apache2 listener directive by @lotharsm in #628
  • build(deps): bump github/codeql-action from 3.28.18 to 3.28.19 in the github-actions group by @dependabot in #635
  • Adds ability to toggle off stripping of private addrs from XFF by @dchandekstark in #619
  • Make progress bar styling more compatible (UXP, etc) by @Fierelier in #636
  • feat(lib): implement request weight by @Xe in #621
  • Update known-instances.md to include SquirrelJME by @XerTheSquirrel in #643
  • fix(anubis): nil check policy loading by @JasonLovesDoggo in #645
  • test: introduce SSH based CI for non-native test hosts by @Xe in #644
  • build(deps): bump github.com/cloudflare/circl from 1.6.0 to 1.6.1 by @dependabot in #650
  • test(ssh-ci): re-enable GOARCH=ppc64le by @Xe in #651
  • fix(gitattributes): update pattern for generated files by @JasonLovesDoggo in #652
  • fix(ci): conditionally run SSH jobs for TecharoHQ/anubis by @JasonLovesDoggo in #654
  • feat: add a strip-base-prefix option by @JasonLovesDoggo in #655
  • refactor(ogtags): optimize URL construction and memory allocations by @JasonLovesDoggo in #647
  • docs(known-instances): add bugs.scummvm.org and gitlab.postmarketos.org by @lotharsm in #661
  • feat: add robots2policy CLI to convert robots.txt to Anubis CEL by @JasonLovesDoggo in #657
  • Add ReactOS to known-instances.md by @ColinFinck in #664
  • build(deps): bump the github-actions group with 3 updates by @dependabot in #666
  • Add the blog section back by @Xe in #670
  • feat: implement a client for Thoth, the IP reputation database for Anubis by @Xe in #637
  • chore(sponsors): update canine.tools logo by @hyperdefined in #672
  • perf: Replace internal SHA256 hashing with xxhash for 4-6x performance improvement by @JasonLovesDoggo in #676
  • perf: replace cidranger with bart for significant performance improvements by @JasonLovesDoggo in #675
  • docs(known-instances): add wiki.koha-community.org by @prettysunflower in #683
  • chore: remove duplicate CHANGELOG entry by @JasonLovesDoggo in #684
  • fix(geo): correct typo "counties" to "countries" by @hydrargyrum in #678
  • docs(known-instances): add extensions.typo3.org by @lotharsm in #691
  • Bump ai.robots.txt to v1.37 by @Dryusdan in #689
  • feat(config): custom weight thresholds via CEL by @Xe in #688
  • fix(config): actually load threshold config by @Xe in #696
  • feat: add default OpenGraph tags to configuration file by @Xe in #694
  • Makefile: Build robots2policy by @heftig in #699
  • fix(default-config): off-by-one error in the default thresholds by @Xe in #701
  • feat: implement imprint/impressum support by @Xe in #706
  • fix(web/js): broken progress bar with slow algo by @yut23 in #673
  • build(deps): bump the github-actions group with 3 updates by @dependabot in #708
  • fix(lib): fix invalid response after success in Chrome by @Xe in #711
  • build(deps): bump the gomod group with 4 updates by @dependabot in #709
  • Populate OpenGraph configurations to Options.OpenGraph by @outloudvi in #717
  • chore: Remove unused/dead code by @JasonLovesDoggo in #703
  • feat: Add option to use HS512 secret for JWT instead of ED25519 by @Earl0fPudding in #680
  • remove incorrect module mentions by @msporleder in #687
  • feat: dynamic cookie domains by @Xe in #722

New Contributors

Full Changelog: v1.19.1...v1.20.0

Contributors

dchandekstark, heftig, and 18 other contributors
Assets 30
Loading
0 Join discussion