This tool has remained uncommitted in my development tree for almost a year. Just minor polish and commit. It can be used to convert some geolocation IP lists to ACLs.
The hash of IPv6 addresses was not properly aligned and resulted in the last quarter of the address not being hashed. In practice, this is rarely detected since MAC addresses are used in the second half. But this becomes very visible with IPv6-mapped IPv4 addresses such as ::FFFF:22.214.171.124 where the IPv4 part is never hashed. This bug has been there forever, since introduction of "balance source" in v1.2.11. The fix must then be backported to all stable versions. Thanks to Alex Markham for reporting this issue to the list !
Possible zero-pointer deference in sess_log(). Checks of return values in sess_log() fix the issue. Fix bad computation in logformat_write_string(). This issue is 1.5-specific and was introduced just before 1.5-dev8. No backport is needed.
Released version 1.5-dev8 with the following main changes : - MINOR: patch for minor typo (ressources/resources) - MEDIUM: http: add support for sending the server's name in the outgoing request - DOC: mention that default checks are TCP connections - BUG/MINOR: fix options forwardfor if-none when an alternative header name is specified - CLEANUP: Make check_statuses, analyze_statuses and process_chk static - CLEANUP: Fix HCHK spelling errors - BUG/MINOR: fix typo in processing of http-send-name-header - MEDIUM: log: Use linked lists for loggers - BUILD: fix declaration inside a scope block - REORG: log: split send_log function - MINOR: config: Parse the string of the log-format config keyword - MINOR: add ultoa, ulltoa, ltoa, lltoa implementations - MINOR: Date and time fonctions that don't use snprintf - MEDIUM: log: make http_sess_log use log_format - DOC: log-format documentation - MEDIUM: log: use log_format for mode tcplog - MEDIUM: log-format: backend source address %Bi %Bp - BUG/MINOR: log-format: fix %o flag - BUG/MEDIUM: bad length in log_format and __send_log - MINOR: logformat %st is signed - BUILD/MINOR: fix the source URL in the spec file - DOC: acl is http_first_req, not http_req_first - BUG/MEDIUM: don't trim last spaces from headers consisting only of spaces - MINOR: acl: add new matches for header/path/url length - BUILD: halog: make halog build on solaris - BUG/MINOR: don't use a wrong port when connecting to a server with mapped ports - MINOR: remove the client/server side distinction in SI addresses - MINOR: halog: add support for matching queued requests - DOC: indicate that cookie "prefix" and "indirect" should not be mixed - OPTIM/MINOR: move struct sockaddr_storage to the tail of structs - OPTIM/MINOR: make it possible to change pipe size (tune.pipesize) - BUILD/MINOR: silent a build warning in src/pipe.c (fcntl) - OPTIM/MINOR: move the hdr_idx pools out of the proxy struct - MEDIUM: tune.http.maxhdr makes it possible to configure the maximum number of HTTP headers - BUG/MINOR: fix a segfault when parsing a config with undeclared peers - CLEANUP: rename possibly confusing struct field "tracked" - BUG/MEDIUM: checks: fix slowstart behaviour when server tracking is in use - MINOR: config: tolerate server "cookie" setting in non-HTTP mode - MEDIUM: buffers: add some new primitives and rework existing ones - BUG: buffers: don't return a negative value on buffer_total_space_res() - MINOR: buffers: make buffer_pointer() support negative pointers too - CLEANUP: kill buffer_replace() and use an inline instead - BUG: tcp: option nolinger does not work on backends - CLEANUP: ebtree: remove a few annoying signedness warnings - CLEANUP: ebtree: clarify licence and update to 6.0.6 - CLEANUP: ebtree: remove 4-year old harmless typo in duplicates insertion code - CLEANUP: ebtree: remove another typo, a wrong initialization in insertion code - BUG: ebtree: ebst_lookup() could return the wrong entry - OPTIM: stream_sock: reduce the amount of in-flight spliced data - OPTIM: stream_sock: save a failed recv syscall when splice returns EAGAIN - MINOR: acl: add support for TLS server name matching using SNI - BUG: http: re-enable TCP quick-ack upon incomplete HTTP requests - BUG: proto_tcp: don't try to bind to a foreign address if sin_family is unknown - MINOR: pattern: export the global temporary pattern - CLEANUP: patterns: get rid of pattern_data_setstring() - MEDIUM: acl: use temp_pattern to store fetched information in the "method" match - MINOR: acl: include pattern.h to make pattern migration more transparent - MEDIUM: pattern: change the pattern data integer from unsigned to signed - MEDIUM: acl: use temp_pattern to store any integer-type information - MEDIUM: acl: use temp_pattern to store any address-type information - CLEANUP: acl: integer part of acl_test is not used anymore - MEDIUM: acl: use temp_pattern to store any string-type information - CLEANUP: acl: remove last data fields from the acl_test struct - MEDIUM: http: replace get_ip_from_hdr2() with http_get_hdr() - MEDIUM: patterns: the hdr() pattern is now of type string - DOC: add minimal documentation on how ACLs work internally - DOC: add a coding-style file - OPTIM: halog: keep a fast path for the lines-count only - CLEANUP: silence a warning when building on sparc - BUG: http: tighten the list of allowed characters in a URI - MEDIUM: http: block non-ASCII characters in URIs by default - DOC: add some documentation from RFC3986 about URI format - BUG/MINOR: cli: correctly remove the whole table on "clear table" - BUG/MEDIUM: correctly disable servers tracking another disabled servers. - BUG/MEDIUM: zero-weight servers must not dequeue requests from the backend - MINOR: halog: add some help on the command line - BUILD: fix build error on FreeBSD - BUG: fix double free in peers config error path - MEDIUM: improve config check return codes - BUILD: make it possible to look for pcre in the default system paths - MINOR: config: emit a warning when 'default_backend' masks servers - MINOR: backend: rework the LC definition to support other connection-based algos - MEDIUM: backend: add the 'first' balancing algorithm - BUG: fix httplog trailing LF - MEDIUM: increase chunk-size limit to 2GB-1 - BUG: queue: fix dequeueing sequence on HTTP keep-alive sessions - BUG: http: disable TCP delayed ACKs when forwarding content-length data - BUG: checks: fix server maintenance exit sequence - BUG/MINOR: stream_sock: don't remove BF_EXPECT_MORE and BF_SEND_DONTWAIT on partial writes - DOC: enumerate valid status codes for "observe layer7" - MINOR: buffer: switch a number of buffer args to const - CLEANUP: silence signedness warning in acl.c - BUG: stream_sock: si->release was not called upon shutw() - MINOR: log: use "%ts" to log term status only and "%tsc" to log with cookie - BUG/CRITICAL: log: fix risk of crash in development snapshot - BUG/MAJOR: possible crash when using capture headers on TCP frontends - MINOR: config: disable header captures in TCP mode and complain
These symbols are only used inside src/checks.c
In order to help users fix their configs, report a warning when a capture has been set on a non-HTTP frontend. This should be backported to 1.4.
Olufemi Omojola provided a config and a core showing a possible crash when captures are configured on a TCP-mode frontend which branches to an HTTP backend. The reason is that being in TCP mode, the frontend does not allocate capture pools for the request, but the HTTP backend tries to use them and dies on the NULL. While such a config has long been unlikely to happen, it looks like people using websocket tend to do this more often now. Change the control to use the pointer instead of the number of captures to know when to log. This bug was reported in 1.4.20, so it must be backported there.
The main stats page says "ressources" (French spelling) rather than "resources" (English spelling). One little patch attached (against v1.4.20). Many thanks, Adrian
__send_log(): the size of the buffer sent is wrong when the facility is lower than 3 digits. logformat_write_string(): computation of size is wrong Note: this was introduced after 1.5-dev7, no backport needed.
Commit a1cc38 introduced a regression which was easy to trigger till ad4cd58 (snapshots 20120222 to 20120311 included). The bug was still present after that but harder to trigger. The bug is caused by the use of two distinct log buffers due to intermediary changes. The issue happens when an HTTP request is logged just after a TCP request during the same second and the HTTP request is too large for the buffer. In this case, it happens that the HTTP request is logged into the TCP buffer instead and that length controls can't detect anything. Starting with bddd4f, the issue is still possible when logging too large an HTTP request just after a send_log() call (typically a server status change). We owe a big thanks to Sander Klein for testing several snapshots and more specifically for taking significant risks in production by letting the buggy version crash several times in order to provide an exploitable core ! The bug could not have been found without this precious help. Thank you Sander ! This fix does not need to be backported, it did not affect any released version.
…cookie The difference could be seen when logging a request in HTTP mode with option tcplog, as it would keep emitting 4 chars. Better use two distinct flags to clear the confusion.
%Bi return the backend source IP %Bp return the backend source port Add a function pointer in logformat_type to do additional configuration during the log-format variable parsing.
Merge http_sess_log() and tcp_sess_log() to sess_log() and move it to log.c A new field in logformat_type define if you can use a logformat variable in TCP or HTTP mode. doc: log-format in tcp mode Note that due to the way log buffer allocation currently works, trying to log an HTTP request without "option httplog" is still not possible. This will change in the near future.
The ->release function of the stream interface is never called upon a shutw() because it's placed after a return statement. It is possible that it has impacted inter-process stick-table replication by preventing a full resync after certain sequences of connection breakage. Since this bug has been present since the introduction of the ->release() callback, it cannot have caused regressions, just possibly non-working situations. This was detected at Exceliance by Emeric Brun during a code review. It is 1.5-specific.
A number of offset computation functions use struct buffer* arguments and return integers without modifying the input. Using consts helps simplifying some operations in callers.
…AIT on partial writes The flags are one-shot but should be maintained over all send() operations as long as send_max is not flushed. The flags were incidentely cleared once a complete send() was performed, regardless of the fact that the send() might have been on the first half of a buffer before a wrapping. The result is that on wrapping data (eg: which happens often with chunked encoding), many incomplete segments are transmitted instead of being aggregated. The fix consists in only flushing the flags only once send_max is empty, which was the expected behaviour. This fix should be backported to 1.4 though it is not critical, just sub-optimal.
Recent commit 62c3be broke maintenance mode by fixing srv_is_usable(). Enabling a disabled server would not re-introduce it into the farm. The reason is that in set_server_up(), the SRV_MAINTAIN flag is still present when recounting the servers. The flag was removed late only to adjust a log message. Keep a copy of the old flag instead and update SRV_MAINTAIN earlier. This fix must also be backported to 1.4 (but no release got the regression).
Commits 5c6209 and 072930 were aimed at avoiding undesirable PUSH flags when forwarding chunked data, but had the undesired effect of causing data advertised by content-length to be affected by the delayed ACK too. This can happen when the data to be forwarded are small enough to fit into a single send() call, otherwise the BF_EXPECT_MORE flag would be removed. Content-length data don't need the BF_EXPECT_MORE flag since the low-level forwarder already knows it can safely rely on bf->to_forward to set the appropriate TCP flags. Note that the issue is only observed in requests at the moment, though the later introduction of server-side keep-alive could trigger the issue on the response path too. Special thanks to Randy Shults for reporting this issue with a lot of details helping to reproduce it. The fix must be backported to 1.4.
When a request completes on a server and the server connection is closed while the client connection stays open, the HTTP engine releases all server connection slots and scans the queues to offer the connection slot to another pending request. An issue happens when the released connection allows other requests to be dequeued : may_dequeue_tasks() relies on srv->served which is only decremented by sess_change_server() which itself is only called after may_dequeue_tasks(). This results in no connection being woken up until another connection terminates so that may_dequeue_tasks() is called again. This fix is minimalist and only moves sess_change_server() earlier (which is safe). It should be reworked and the code factored out so that the same occurrence in session.c shares the same code. This bug has been there since the introduction of option-http-server-close and the fix must be backported to 1.4.
Since commit 115acb9, chunk size was limited to 256MB. There is no reason for such a limit and the comment on the code suggests a missing zero. However, increasing the limit past 2 GB causes trouble due to some 32-bit subtracts in various computations becoming negative (eg: buffer_max_len). So let's limit the chunk size to 2 GB - 1 max.
commit a1cc381 introduced an undesirable \0\n ending on HTTP log messages. This is because of an extra character count passed to __send_log() which causes the LF to be appended past the \0. Some syslog daemons thus log an extra empty line. The fix is obvious. Fix the function comments to remind what they expect on their input. This is past 1.5-dev7 regression so there's no backport needed.
The principle behind this load balancing algorithm was first imagined and modeled by Steen Larsen then iteratively refined through several work sessions until it would totally address its original goal. The purpose of this algorithm is to always use the smallest number of servers so that extra servers can be powered off during non-intensive hours. Additional tools may be used to do that work, possibly by locally monitoring the servers' activity. The first server with available connection slots receives the connection. The servers are choosen from the lowest numeric identifier to the highest (see server parameter "id"), which defaults to the server's position in the farm. Once a server reaches its maxconn value, the next server is used. It does not make sense to use this algorithm without setting maxconn. Note that it can however make sense to use minconn so that servers are not used at full load before starting new servers, and so that introduction of new servers requires a progressively increasing load (the number of servers would more or less follow the square root of the load until maxconn is reached). This algorithm ignores the server weight, and is more beneficial to long sessions such as RDP or IMAP than HTTP, though it can be useful there too.
…based algos The leastconn algorithm should be of kind "connection-based", not "leastconn" if we want to later support other connection-based LB algos.
When a "listen" instance uses a "default_backned" rule and has servers, the servers will never be used. Report it so that users don't get trapped.
If running "make PCREDIR=" will not force to add -I nor -L anymore.
Also move human_time() to standard.c since it's not related to timeval calculations.
Implementations that write result from left to right
parse_logformat_string: parse the string, detect the type: text, separator or variable parse_logformat_var: dectect variable name parse_logformat_var_args: parse arguments and flags add_to_logformat_list: add to the logformat linked list