Add JAWS v2 to jax_fingerprint#13073
Closed
bneradt wants to merge 904 commits into
Closed
Conversation
* traffic_ctl - Implement server status command option.
With the removal of traffic_manager we lost this command option, so now
this change makes ATS to respond the rpc call with some basic `TSSystemState`
Information, like intimidation status, draining status, etc.
The output is a simple json output.
Command option is read only, no change is done in the internal ATS state.
Example:
$ traffic_ctl server status | jq
{
"initialized_done": "true",
"is_ssl_handshaking_stopped": "false",
"is_draining": "false",
"is_event_system_shut_down": "false"
}
This also includes some autests to validate the output. Changes to
existing traffic_ctl autest machinery is made to make it simpler to
extend and add more tests.
* Fix docs
(cherry picked from commit 425939c)
…pache#12274) (apache#1065) For directory changes, we compared the event's name (which indicates which file in the directory changed) against the config's basename but limited the check to basename_len via strncmp. Unwittingly, though, if event->name was more bytes than basename_len, we could incorrectly match on the substring. For instance, with a config like this: /acme /tmp/sb/healthchecks/acme text/plain 200 404 /acme-ssl /tmp/sb/healthchecks/acme-ssl text/plain 200 404 An event for acme-ssl changes would accidentally match for acme, because acme's basename_len is 4, while event->name is acme-ssl, thus strncmp on the event would incorrectly indicate a match because the first four bytes were the same for each. This extracts event/config file matching to a function so it is easier to understand and ensures that event->name and basename are the same length to avoid the accidental match. This also adds an autest for healthchecks which reproduces the above issue and verifies basic file event processing functionality. (cherry picked from commit 482e0c6)
UUID values are 128 bit values designed to be univerally unique identifiers across all practical uses and environments. Snowflake ID values are 64 bit values that are designed with a more limited goal: to be unique across all values in a single organization. By limiting their scope, they limit their size which is helpful for performance and storage needs. This updates the connection id to use SnowflakeID values so that they are unique across reboots and typically unique across ATS instances in a cdn. This also adds psfid, a logging field similar to puuid, but that logs the process's snowflake ID instead of its UUID.
…#1067) Add max inclusion depth support to esi plugin --------- Co-authored-by: Shu Kit Chan <kichan@c40b7d47.home-kichan.bf2.ows.oath.cloud>
- Adds a ASCII table for URL parts - Order the table by presence in url from left to right - Add example value for each (cherry picked from commit 7a3486a)
* Add max inclusion depth support to esi plugin * fix formatting (cherry picked from commit 7f178de)
This should fix build warnings.
Our TLS CLIENT_HELLO processing logic assumed all CLIENT_HELLO bytes came in a single TCP packet. However, with more recent cryptographic ciphers, the CLIENT_HELLO is often greater than the standard 1,500 byte MTU, so the CLIENT_HELLO is being delivered in multiple packets. This updates our logic to properly buffer and parse data across multiple socket reads. Fixes: apache#11758 Co-authored-by: Brian Neradt <brian.neradt@gmail.com>
This pulls in the other changes from Masakazu's 10.1.x PR: apache#12303 I started from 10.0.0_76, applied his patch from the above, then pulled in all changes. So our release is now identical to his 10.1.x version with this patch.
* stats_over_http: Add Prometheus format option This adds the Prometheus format as an optional output to the stats_over_http plugin. This also adds /json, /csv, and /prometheus path suffix support in case using the Accept header is problematic. * Use underscores for characters Prometheus doesn't like
This cherry picks a review comment update from: apache#12290
…he#12257) (apache#1074) * HTTP/2: Wait END_STREAM flag on half-closed(local) stream state * Actively close stream in half-closed(local) state * Add AuTest * Fix format of AuTests * Address comments (cherry picked from commit c3a40a7) Co-authored-by: Masaori Koshiba <masaori@apache.org>
…#12293) (apache#12303) * Add a setting to choose the data source of IP address for ACL * Address compile errors * Address more issues * Add missing files (cherry picked from commit 91a654d) Conflicts: src/proxy/http2/Http2SessionAccept.cc
This fixes the handling of the server processes in per_server_connection_max.test.py. Before this change: ``` Running Test per_server_connection_max:Warning: Overriding process object dns2 Warning: Overriding process object server2 ``` Addressing this requires making sure each of those proesses are started with different names. (cherry picked from commit 504f4a3)
Our TLS CLIENT_HELLO processing logic assumed all CLIENT_HELLO bytes came in a single TCP packet. However, with more recent cryptographic ciphers, the CLIENT_HELLO is often greater than the standard 1,500 byte MTU, so the CLIENT_HELLO is being delivered in multiple packets. This updates our logic to properly buffer and parse data across multiple socket reads. Fixes: apache#11758 (cherry picked from commit 572ec5d)
…on (apache#12278) * Update url_m_loc variable to the newly generated on in set-destination * Update operators.cc * Update operators.cc * Update operators.cc (cherry picked from commit 02139d5)
TLS key exchange terminology has shifted from curves to groups since future cryptographic mechanisms may not be eliptic curve (such as post quantum KEM). cqssg is a new log field being added to print the TLS group name for client side connections. The documentation is also updated to encourage uses to, long term, phase out the use of cqssu in favor of cqssg. (cherry picked from commit d09b809)
* unique_ptr<char[]> -> array; default initialize .value * SnowflakeIDGenerator This encapsulates the state for snowflake id generation in a generator, avoiding global mutable state. --------- Co-authored-by: Brian Neradt <brian.neradt@gmail.com>
There were no conflicts.
This is a cherry-pick of: apache#12325 --- This addresses the following assertion crash in HttpSM::attach_server_session: hsm_release_assert(server_entry == nullptr); The history shows this: ``` $1 = History(capacity=65, history_pos=118, num_entries=65) = { HttpSM.cc:5018 do_cache_lookup_and_read TS_EVENT_NONE (0) reent= 22, HttpSM.cc:844 state_watch_for_client_abort TS_EVENT_VCONN_READ_READY (100) reent= 1, HttpSM.cc:844 state_watch_for_client_abort TS_EVENT_VCONN_READ_READY (100) reent= 1, HttpSM.cc:844 state_watch_for_client_abort TS_EVENT_VCONN_READ_READY (100) reent= 1, HttpSM.cc:844 state_watch_for_client_abort TS_EVENT_VCONN_READ_READY (100) reent= 1, HttpSM.cc:844 state_watch_for_client_abort TS_EVENT_VCONN_READ_READY (100) reent= 1, HttpSM.cc:2081 state_send_server_request_header TS_EVENT_VCONN_WRITE_COMPLETE (103) reent= 1, HttpSM.cc:6354 do_setup_client_request_body_tunnel NO_EVENT (34463) reent= 1, HttpSM.cc:3897 tunnel_handler_post_server TS_EVENT_ERROR (3) reent= 0, HttpSM.cc:2735 tunnel_handler_post TS_EVENT_HTTP_TUNNEL_EVENT_DONE (2301) reent= 1, HttpSM.cc:5979 handle_post_failure TS_EVENT_NONE (0) reent= 1, HttpCacheSM.cc:104 state_cache_open_read TS_EVENT_CACHE_OPEN_READ_FAILED (1103) reent= -31073, HttpSM.cc:2561 state_cache_open_read TS_EVENT_CACHE_OPEN_READ_FAILED (1103) reent= 7, HttpSM.cc:2221 add_to_existing_request NO_EVENT (34463) reent= 11, HttpSM.cc:5018 do_cache_lookup_and_read TS_EVENT_NONE (0) reent= 6, HttpSM.cc:1769 state_http_server_open TS_EVENT_VCONN_EOS (104) reent= 1, HttpSM.cc:1769 state_http_server_open Unknown event 1400 (1400) reent= 1} ``` Notice that an error occurs in tunnel_handler_post_server with the expectation that tunnel_handler_post would handle it. But tunnel_handler_post only handled UA failures, not SERVER ones. This updates it to handle server failures too. Fixes: apache#12240 Co-authored-by: Brian Neradt <brian.neradt@gmail.com>
…he#12326) The issue was related to the files that aren't associated with a record name. The code wasn't taking into consideration that there are files that are bound to a parent file and not to a record. This fixes the issue and avoid reporting misleading errors to the rpc caller. Fixes issue apache#12262 (cherry picked from commit 24ebd0f)
…ache#12327) Beside errors in the logic which was already covered this also marks the return code to error when the RPC sends back a response with some sort of failure. With this change traffic_ctl callers can now validate the exit code also considering the RPC errors/failures. (cherry picked from commit 5e334de)
Boxes with OCSP configured typically have their logs containing many of these: ``` $ tail diags.log [Jul 1 18:14:05.654] [ET_OCSP 0] NOTE: OCSP refresh started [Jul 1 18:14:05.654] [ET_OCSP 0] NOTE: OCSP refresh finished [Jul 1 18:15:05.649] [ET_OCSP 0] NOTE: OCSP refresh started [Jul 1 18:15:05.649] [ET_OCSP 0] NOTE: OCSP refresh finished [Jul 1 18:16:05.648] [ET_OCSP 0] NOTE: OCSP refresh started [Jul 1 18:16:05.648] [ET_OCSP 0] NOTE: OCSP refresh finished [Jul 1 18:17:05.644] [ET_OCSP 0] NOTE: OCSP refresh started [Jul 1 18:17:05.644] [ET_OCSP 0] NOTE: OCSP refresh finished [Jul 1 18:18:05.640] [ET_OCSP 0] NOTE: OCSP refresh started [Jul 1 18:18:05.640] [ET_OCSP 0] NOTE: OCSP refresh finished ``` This is probably not generally helpful. There are error messages for OCSP failure scenarios. This patch downgrades these to debug logs. (cherry picked from commit 0f50772)
* Expose JSON client timeout to CLI * Make default timeout much larger (cherry picked from commit 624462f)
- Copy a whole pp_info for logging - Calculate the length with alignment in consideration - Put null terminator at the end of the log field value (cherry picked from commit 24e0170)
Add a new HTTP stat to track 429 (Too Many Requests) responses sent to clients. This is useful for monitoring rate-limiting behavior, as the rate_limit plugin defaults to returning 429 when limits are exceeded. The metric follows the same pattern as existing per-status-code counters (400-416) across HttpConfig, HttpTransact, traffic_top, and traffic_logstats. Also enhance the ATSReplayTest framework with a metric_checks feature that allows replay YAML files to declare expected metric values. After traffic completes, follow-up test runs automatically verify the metrics via traffic_ctl. This required creating the ATS process on the Test object (rather than a TestRun) when post-traffic verification is needed, so that ATS persists across test runs.
…e#12760) (apache#1155) The probe-full-json transform called TSContCall(WRITE_READY) even when no data was consumed (towrite == 0), causing a tight loop that starved other transactions on the same ET_NET thread. Fix: Only call TSVIOReenable/TSContCall when towrite > 0. Added autest with slow chunked origin to verify the fix. (cherry picked from commit c3bcdf0) Co-authored-by: Bryan Call <bcall@apache.org>
…he#1156) difference_msec only checked whether ms_end was 0 (unset) before computing the difference. When ms_start was unset (0) but ms_end was set, the subtraction produced a raw epoch value (~13-digit number in milliseconds) instead of returning the "missing" sentinel (-1). This affects msdms log fields where the start milestone is never set for certain transaction types -- for example, origin timing fields (o_conn, o_wait, o_hdr, o_proc) on cache hits, or TLS fields on non-TLS connections. Instead of logging -1, these fields logged enormous epoch values. Fix: check both ms_start and ms_end for 0 before computing the difference. Note: some fields will still show epoch-scale values by design -- o_proc and c_xfer on cache hits, and o_wait/dns on connection-reuse transactions -- because both milestones are set but span across the full request lifecycle. This is expected and documented.
ATS uses @ headers as a mechanism for internal communication among plugins. They are not allowed by the RFC and should not appear externally, either from the client or server nor to the client or server. This therefore strips client request and origin response @ headers before READ_REQUEST_HDR and READ_RESPONSE_HDR hooks run so plugins cannot observe such internal headers from untrusted peers which may confuse the plugins. This also extends ATSReplayTest replay handling so tests can stage custom plugins and pass structured plugin arguments directly from replay configuration.
Log and count when ATS strips untrusted @ headers from client requests and origin responses before plugin hooks. This gives operators visibility into attempts to inject internal @ metadata. Expand the at_headers replay coverage to verify the new logs and metrics, cover both global and remap plugin contexts with one plugin, and confirm response-side stripping in the remap path.
The filter_body test's Proxy Verifier client can exit with either 0 or 1 depending on timing of connection closure during request body blocking. The ATSReplayTest extension only supported a single scalar return_code value, so the test would intermittently fail when the client happened to exit with 0 instead of the expected 1. Support a list of return codes in the replay YAML by wrapping them in Any() when a list is provided, and update the filter_body test to accept either exit code. (cherry picked from commit 1b6847f)
…ody blocking (apache#12876) * Fix filter_body request body blocking race -- transform blocks after headers are sent to origin, causing Content-Length mismatch and 30s hang. Set 1s origin no-activity timeout on block, add SEND_RESPONSE_HDR hook for deterministic 403. * Conditional SEND_RESPONSE_HDR hook, re-enable response blocking -- only attach hook when blocking a request body. Re-enable response blocking in tests. * Add FILTER_BODY_BLOCK_CANARY to prove blocked body never reaches origin -- canary prefix in blocked body + excludes rule on verifier server output. * Add server/client log_validation to ats_replay.test.ext -- remove confusing traffic_out node, use contains/excludes directly. Add client validation support. Fixes: apache#12875 (cherry picked from commit 3bdfaa5)
The prev_is_cr flag in ChunkedHandler::read_size() was being set within conditional expressions but not consistently updated, which could cause it to retain stale values across parsing iterations. Update the flag at the end of each loop iteration to ensure it accurately reflects the current character state. This improves the correctness of line break validation when parsing chunked transfer encoding with strict_chunk_parsing enabled.
Add a check to make sure we have a valid buffer for tunnel producer to write on. (cherry picked from commit d3ef523)
Conflicts: tests/gold_tests/chunked_encoding/replays/malformed_chunked_header.replay.yaml
…pache#1163) * got client hello routed to plugins * Creates ja4 fingerprint with boringssl * cleanup a bit * make ssl_client_hello const * spaces cleanup * cleanup code * more cleanup * Update plugin.cc * Update plugin.cc * Update ts.h * Update apidefs.h.in * Update to make more clean * Update data * address comments * Update ja4_fingerprint.en.rst * Add docs * Update TSVConnClientHelloGet.en.rst * Update TSVConnClientHelloGet.en.rst * Address comments * Update TSClientHello.en.rst * Update TSClientHello.en.rst * Eliminate heap allocations * Address Copilot comments * Address Copilot comments * Update doxygen comments --------- Co-authored-by: Jasmine Emanouel <jasmine.nahrain@gmail.com> (cherry picked from commit 3f6f207) Co-authored-by: Masakazu Kitajo <maskit@apache.org>
See: https://github.com/ATSUnderground/trafficserver/pull/142 (cherry picked from commit 837425e4c59e0c90fb199ac8aa3d888817bb4ea5) Co-authored-by: Serris Santos <lserris@apache.org>
apache#1160) Add a check to make sure we have a valid buffer for tunnel producer to write on. (cherry picked from commit 895878685bb8877dd39c5c0401a080178ccac70b) Co-authored-by: Damian Meden <dmeden@apache.org>
Import the experimental jax_fingerprint plugin from PR 12995 as a single parent commit for the local JAWS v2 work. The plugin sources, JA tests, and supporting build files are kept identical to the current PR branch so future syncs and reviews stay straightforward. This is a clean cherry-pick of: apache#12995 (cherry picked from commit 910f3e5)
This keeps the local experimental plugin roster intact when the jax_fingerprint import is applied on top of the 10.0.x tree. The imported parent commit also pulled in CMake entries for connection_exempt_list and realip, but those plugin directories do not exist in this checkout and they break a plain build_ats_ut configure. This restores abuse_shield for the local tree and drops the missing plugin references so the parent commit configures cleanly before the JAWS work.
Extend jax_fingerprint with modular JA3-family methods for JA3, JA3 raw, GREASE-preserved JA3 raw, JAWS, and JAWS v2. The new JA3 summary and encoder modules keep the method adapters thin, preserve the frozen JA3 and JAWS v1 outputs, and add JAWS v2 without disturbing the generic plugin core. Multiple plugin instances can also share context and log handles so operators can emit these fingerprints side by side.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
19 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Extend jax_fingerprint with modular JA3-family methods for
JA3, JA3 raw, GREASE-preserved JA3 raw, JAWS, and JAWS v2.
The new JA3 summary and encoder modules keep the method
adapters thin, preserve the frozen JA3 and JAWS v1 outputs,
and add JAWS v2 without disturbing the generic plugin core.
Multiple plugin instances can also share context and log
handles so operators can emit these fingerprints side by side.