New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Load balancing #5

Closed

krizhanovsky opened this issue Sep 6, 2014 · 3 comments

Labels

Milestone

0.3.0 Proxy Filter

Contributor

krizhanovsky commented Sep 6, 2014

Following sched modules must be implemented:

round-robin
by list of statically specified URL, Host or backed IP addresses (i.e. each backend server must have it's own list of URLs and/or Host values (or IP addresses instead of DNS names) by which are sent to it). This is very similar to location based load balancing of Nginx;
by hash calculated on URI, Host HTTP header or client IP

See Nginx load balancing as an example.

The system must ensure that only one shceduler module can be loaded by a user.

Tempesta must add X-Forwarded-For HTTP header with the client IP address to forwarded to backed server request.

Failovering over backend nodes also must be implemented. See HAProxy

The text was updated successfully, but these errors were encountered:

krizhanovsky added the enhancement label

krizhanovsky assigned vdmit11

krizhanovsky mentioned this issue

Merged

vdmit11 added a commit that referenced this issue


          Fix small code review issues

a5bf0d2

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Add TfwStr comparison functions

5eb6fcf

1.Add new functions for comparing TfwStr with ANSI strings.

2. Extract TfwStr code to a separate .c file and cover it with unit
tests.

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Basic tfw_sched_match implementation

0754b4a

- Scheduler modules are moved to sub-directories to group their files.
- tfw_sched_match implements "prefix" and "equal" comparisons.
- tfw_sched_match has a sysctl interface with a tiny config parser.

Related to task:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Fix stylistic issues.

f076c68

Related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Move lib/ files back to the root directory of the tempesta_fw module

31aecb1

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Add TfwStr comparison functions

b2d02e2

1.Add new functions for comparing TfwStr with ANSI strings.

2. Extract TfwStr code to a separate .c file and cover it with unit
tests.

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Basic tfw_sched_match implementation

900631a

- Scheduler modules are moved to sub-directories to group their files.
- tfw_sched_match implements "prefix" and "equal" comparisons.
- tfw_sched_match has a sysctl interface with a tiny config parser.

Related to task:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Fix stylistic issues.

aa87bd7

Related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Move lib/ files back to the root directory of the tempesta_fw module

8f38023

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Fix stylistic issues noted on the latest review

f74e132

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          Extract unit tests to a separate module

75d3221

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          partially extract http_match logic to a separate module

a7d58a3

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          add basic tests for http uri/host/header matching

49a119e

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          migrate from array to linked list for http matching rules

d7f171c

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          migrate tfw_sched_match to the new http_match API

a062b86

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          implement tfw_sched_http

e5ef3af

This large commit is combined from a series of previous small commits
that contain basic implementation for tfw_sched_http - the new
rule-based scheduler.

Changes overview:
 - Add simple unit testing framework.
   Tests run in kernel space in a separate module 'tfw_test'.
 - Extract TfwStr to a separate unit, implement matching functions,
   cover them with unit tests.
 - Extract TfwAddr to a separate unit, lib.c doesn't exist anymore.
 - Add http_match.c - the HTTP Request matching logic.
 - Add tfw_sched_http.c - the rule-based scheduler.

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          add tests for tfw_str_add_compound()

f1f1d3d

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          fix stylistic issues related to tfw_sched_http

59d2d6d

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          remove duplicate logic from TfwStr comparison functions

e61e7b3

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          implement functions for matching key-value pairs in TfwStr strings

a5ee1d2

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          implement HTTP header matching

ef30e82

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          remove redundant TfwStr comparison functions

0aea53b

related to: #5 - Load balancing

vdmit11 added a commit that referenced this issue


          optimize tfw_str_eq_kv()

07c31db

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          add more tests for http_match.c logic

c259d23

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          update tfw_sched_http according to the new http_match.c logic

6ab0144

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          remove redundant tfw_http_hdr_name()/tfw_http_hdr_id() functions

7b5adee

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          basic tfw_sched_hash implementation

fdd1a99

related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          fix stylistic issues

8f20f00

related to:
#5 - Load balancing

vdmit11 mentioned this issue

basic tfw_sched_hash implementation #27

Merged

vdmit11 added a commit that referenced this issue


          Fix review issues

7ef0295

Related to:
#5 - Load balancing

vdmit11 added a commit that referenced this issue


          fix RFC references in URI and Host matching code

982d66d

related to:
#5 - Load balancing

Contributor

vdmit11 commented Nov 13, 2014

TBD:

Add the X-Forwarded-For header to requests.
finish the "failover" branch.
The failover implies that there will be more than a single connection per client/server.
There are some simple architectural changes required, in particular:
1. TfwSession should link TfwConnection objects instead of TfwClient and TfwServer.
2. TfwServer should contain a pool of connections instead of just one connection.
  Partially they are done, but now there are some circular dependencies (TfwConnection depends on TfwClient and vise versa), they should be solved before finishing the issue.

vdmit11 mentioned this issue

fix defects introduced by commit 1cd52cf #23

Closed

vdmit11 added a commit that referenced this issue


          Implement back-end server failover.

f995a92

The change introduces a small set of parallel connections to each
TfwServer. They are switched in a round-robin manner when a message
is scheduled. When a connection is closed, another is chosen and the
closed one is re-connected in background.

Also many connections per server implies that TfwSession doesn't link
TfwClient and TfwServer anymore. Instead, it links client/server sockets.

Related to: #5 - load balancing

vdmit11 added a commit that referenced this issue


          implement basic X-Forwarded-For header support

84f497a

The commit introduces two major things:
1. The HTTP parser is now able to parse the X-Forwarded-For header and
   put it to the headers table, although t doesn't parse IP addresses
   or distinct Node IDs yet.
2. Client's IP address is added to X-Forwarded-For header of each
   forwarded HTTP request. If there is no such header - it is created,
   if it is already there, then the client's IP is appended to the list.

Also the commit contains fixes for the HTTP parser that used to crop
parsed HTTP headers and write them into each other's locations in the
headers table.

Related to: #5 - Load Balancing

vdmit11 mentioned this issue

implement basic X-Forwarded-For header support #33

Merged

vdmit11 added a commit that referenced this issue


          implement basic X-Forwarded-For header support

669b0c8

The commit introduces two major things:
1. The HTTP parser is now able to parse the X-Forwarded-For header and
   put it to the headers table, although t doesn't parse IP addresses
   or distinct Node IDs yet.
2. Client's IP address is added to X-Forwarded-For header of each
   forwarded HTTP request. If there is no such header - it is created,
   if it is already there, then the client's IP is appended to the list.

Also the commit contains fixes for the HTTP parser that used to crop
parsed HTTP headers and write them into each other's locations in the
headers table.

Related to: #5 - Load Balancing

Contributor Author

krizhanovsky commented Jan 20, 2015

tfw_sched_hash must be updated by Rendezvous hashing (http://en.wikipedia.org/wiki/Rendezvous_hashing) for backend servers failover.

Contributor Author

krizhanovsky commented Mar 11, 2015

It seems this part of the task "Failovering over backend nodes also must be implemented. See HAProxy" is not implemented. We need backup nodes in HAProxy terms WAF installations. The typical installation is Tempesta balances traffic among WAF servers and does preliminary filtering and the whole cluster DDoS protection. WAF nodes do high-level heavy weight filtering and pass the traffic to web servers. However, if WAF nodes fail, then Tempesta must bypass traffic directly to backend Web-servers, but it must not send any traffic to the servers in normal circumstances.

krizhanovsky added this to the 0.3.0 Proxy Filter milestone

vdmit11 mentioned this issue

Sticky cookies #29

Closed

krizhanovsky unassigned vdmit11

krizhanovsky closed this as completed

sergsever mentioned this issue

Functional test for Frang #673

Closed

const-t added a commit that referenced this issue


          Added patch from PR #5 and removed unnecessary initializations

279b397

const-t added a commit that referenced this issue


          Added patch from PR #5 and removed unnecessary initializations

b777b84

const-t added a commit that referenced this issue


          Remove pfl optimization (#1772)

f6c0256

Removed pfl optimization __lookup_pgfrag_room

After analyzing perfstat and benchmarks we came to the conclusion,
that pfl not working as expected and looks like overengineering.

The most optimistic case of observer results(Response 128KB):
SS pfl hits	: 2969144
SS pfl misses	: 3944903

* Removed pfl statistics

* Added patch from PR #5 and removed unnecessary initializations

EvgeniiMekhanik mentioned this issue

Kernel panic during reconfiguration #1806

Closed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment