Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

New IPv4 filter implementation that isn't a memory hog -- fixes rtorrent issue #106 and #71 #109

Open
wants to merge 1 commit into from

8 participants

@sallyswiss

New extents.h data structure in libtorrent so that it does not waste memory. it is now an ordered map instead of a crazy sparse hash, much better, still log n access time. it also stores ranges as in p2p file format so it can handle p2p data from bluetack lists as well as cidr notation which is now only thing supported. There is corresponding commit and pull request for rtorrent that makes the necessary changes there too.

@jenkins101

so with this added it is possible to use p2p format files?

with comments?

@sallyswiss

when parsing file line everything up to colon ':' is ignored. if you mean comments that start with '#' then handling is not ideal as parser will still try to parse line so '# abc def: 1.2.3.4.5-1.2.3.4.10' will still get added to list even though it is commented out. However, it will not crash on comments so if nothing that could be parsed as ip address is in comments they will not prevent program from running.

@jenkins101

good!

@jenkins101

mm...

seems like you have pulled from your master...
you should branch then add pull from branch instead...

latest commit is unrelated to this issue.
looks interesting though... :-)

@sallyswiss

ugh. sorry. I am familiar with git but new to github.

do you know what would be the easiest way to fix it now that pull request is sent and second commit is pushed to the same branch?

@jenkins101

yes... did the same thing in the beginning...

you need to do a rebase and then a force push

something like this if you only did one commit i think...
git rebase -i HEAD~1. leave the first line the way it is, change the following line from 'pick' to 'f'. exit the editor and it'll squash them up. force push to your master branch.

but if you want to repull from a branch it is easier to close this delete your repo and refork...
save your work some were though!
create a branch from master add work and then do pull from branch

or you probably could rebase back to beginning...
aa... i think you will solve it... ;-)

@jenkins101

the github guides are a little messy...

I have found this one really useful: https://code.djangoproject.com/wiki/CollaborateOnGithub

@sallyswiss

doing rebase would just combine commits though. not what I want.

instead [1] checkout last commit in rakshasa master [2] create new branch from there [3] cherry-pick latest commit into new branch [4] revert commit in master. problem solved.

@jenkins101

no... you want to do rebase with the f option

f, fixup = like "squash", but discard this commit's log message
it will be removed from your master...
I did it and it worked for me.

then you can branch and add it to a branch but you will still have this commit thats not in rakshasas master yet...
so you probably want to close this and then refork or rebase back to beginning. then branch... and pull again

@jenkins101

or if you can rebase in to a new branch... but i dont know hove to do that...

@jenkins101

I tested it joust now
git rebase -i HEAD~3 leave first line change other 2 to f save and they are gone...

you probably had to do that revert first yes...

@sallyswiss

ok. all fixed now.

@jenkins101

but you should do a rebase to fixup all the commit messages...
so its only one again...

else al commit messages will go into rtorrent...

@sallyswiss

yes. I just did a rebase. only one commit is left.

@jenkins101

a sorry... had to reload the page... <:-)

@Jellyfrog

Now the waiting being for the mighty @rakshasa

@DugieHowsa

Tried the new version with the ipfilter corrections.

Loaded a 5.6 MB CIDR format list. Memory usage appears to be minimal:

1358359195 I Loaded 338529 unwanted address blocks (3967 kb in-memory) from '~/ipfilter.dat'.

Not sure if it is actually blocking though, as I loaded a torrent and let it run for a few minutes, but I didn't see any messages in the logs about connections being blocked.

I'll continue to run the service with logging over the next few days to see if anything pops up.

@rakshasa
Owner

Thanks for the work.

I'll do a more thorough review later, however I'd like to note that C-style comments are not accepted in my codebase.

@sallyswiss

sorry for not matching preferred style. I will switch comments to C++ style.

@sallyswiss sallyswiss New implementation of ip filter that [1] is not a memory hog [2] can …
…handle p2p lists from bluetack [3] still has quick log n lookup time for extents data structure.
666a50a
@sallyswiss

all comments are now changed to C++ style in both rtorrent and libtorrent. I also made two minor improvements. I added code to handle comments "#" when parsing filter file lines and added logging when connection to unwanted peer is prevented. this addresses issue pointed out by DugieHowsa above. wihtout logging it is hard to know when filter is working. log level is set to LOG_PEER_INFO.

@rakshasa if you need more information when reviewing my code please ask and I will be happy to help. if you want any further changes beyond c++ comments just ask.

@jenkins101

Is it not time to implement this and make a new release son...
ip blocking is not working in current rel without this...

I have been run this in my OpenELEC rTorrent service add-on for a while and it seams to work good.

@rakshasa
Owner

Currently not in a position to handle this and other issues that have to be done before a release.

@garmahis

Can you make it compatible with the standard ipfilter.dat files?

@gauthier-delacroix

@rakshasa I'm using @sallyswiss fork for more than 1 year now and it works like a charm with IP filters, without memory leak.

Now that you have bumped the new release, maybe could you merge it ?

@Anthony25

Any news about a possible merge ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Commits on Jan 18, 2013
  1. @sallyswiss

    New implementation of ip filter that [1] is not a memory hog [2] can …

    sallyswiss authored
    …handle p2p lists from bluetack [3] still has quick log n lookup time for extents data structure.
This page is out of date. Refresh to see the latest.
Showing with 211 additions and 91 deletions.
  1. +210 −90 src/command_ip.cc
  2. +1 −1  src/rpc/ip_table_list.h
View
300 src/command_ip.cc
@@ -45,42 +45,17 @@
#include "globals.h"
#include "command_helpers.h"
-void
-ipv4_filter_parse(const char* address, int value) {
- uint32_t ip_values[4] = { 0, 0, 0, 0 };
- unsigned int block = rpc::ipv4_table::mask_bits;
-
- char ip_dot;
- int values_read;
-
- if ((values_read = sscanf(address, "%u%1[.]%u%1[.]%u%1[.]%u/%u",
- ip_values + 0, &ip_dot,
- ip_values + 1, &ip_dot,
- ip_values + 2, &ip_dot,
- ip_values + 3,
- &block)) < 2 ||
-
- // Make sure the dot is included.
- (values_read < 7 && values_read % 2) ||
-
- ip_values[0] >= 256 ||
- ip_values[1] >= 256 ||
- ip_values[2] >= 256 ||
- ip_values[3] >= 256 ||
-
- block > rpc::ipv4_table::mask_bits)
- throw torrent::input_error("Invalid address format.");
- // E.g. '10.10.' will be '10.10.0.0/16'.
- if (values_read < 7)
- block = 8 * (values_read / 2);
- lt_log_print(torrent::LOG_CONNECTION_DEBUG, "Adding ip filter for %u.%u.%u.%u/%u.",
- ip_values[0], ip_values[1], ip_values[2], ip_values[3], block);
+#include <stdio.h>
+#include <string.h>
+#include <stdlib.h>
+#include <unistd.h>
+#include <arpa/inet.h>
+#include <netinet/in.h>
+
+bool ipv4_range_parse(const char* address, uint32_t* address_start, uint32_t* address_end);
- torrent::PeerList::ipv4_filter()->insert((ip_values[0] << 24) + (ip_values[1] << 16) + (ip_values[2] << 8) + ip_values[3],
- rpc::ipv4_table::mask_bits - block, value);
-}
torrent::Object
apply_ip_tables_insert_table(const std::string& args) {
@@ -98,7 +73,8 @@ apply_ip_tables_size_data(const std::string& args) {
if (itr != ip_tables.end())
throw torrent::input_error("IP table does not exist.");
- return itr->table.sizeof_data();
+ uint32_t size = itr->table.sizeof_data();
+ return size;
}
torrent::Object
@@ -110,20 +86,18 @@ apply_ip_tables_get(const torrent::Object::list_type& args) {
const std::string& name = (args_itr++)->as_string();
const std::string& address = (args_itr++)->as_string();
-
- // Move to a helper function, add support for addresses.
- uint32_t ip_values[4];
-
- if (sscanf(address.c_str(), "%u.%u.%u.%u",
- ip_values + 0, ip_values + 1, ip_values + 2, ip_values + 3) != 4)
- throw torrent::input_error("Invalid address format.");
+ uint32_t address_start;
+ uint32_t address_end;
rpc::ip_table_list::iterator table_itr = ip_tables.find(name);
if (table_itr == ip_tables.end())
throw torrent::input_error("Could not find ip table.");
-
- return table_itr->table.at((ip_values[0] << 24) + (ip_values[1] << 16) + (ip_values[2] << 8) + ip_values[3]);
+ if (!ipv4_range_parse(address.c_str(), &address_start, &address_end))
+ throw torrent::input_error("Invalid address format.");
+ if(!table_itr->table.defined(address_start, address_end))
+ throw torrent::input_error("No value defined for specified IP(s).");
+ return table_itr->table.at(address_start, address_end);
}
torrent::Object
@@ -137,14 +111,6 @@ apply_ip_tables_add_address(const torrent::Object::list_type& args) {
const std::string& address = (args_itr++)->as_string();
const std::string& value_str = (args_itr++)->as_string();
- // Move to a helper function, add support for addresses.
- uint32_t ip_values[4];
- unsigned int block = rpc::ipv4_table::mask_bits;
-
- if (sscanf(address.c_str(), "%u.%u.%u.%u/%u",
- ip_values + 0, ip_values + 1, ip_values + 2, ip_values + 3, &block) < 4 ||
- block > rpc::ipv4_table::mask_bits)
- throw torrent::input_error("Invalid address format.");
int value;
@@ -158,8 +124,12 @@ apply_ip_tables_add_address(const torrent::Object::list_type& args) {
if (table_itr == ip_tables.end())
throw torrent::input_error("Could not find ip table.");
- table_itr->table.insert((ip_values[0] << 24) + (ip_values[1] << 16) + (ip_values[2] << 8) + ip_values[3],
- rpc::ipv4_table::mask_bits - block, value);
+ uint32_t address_start;
+ uint32_t address_end;
+ if (ipv4_range_parse(address.c_str(), &address_start, &address_end) )
+ table_itr->table.insert(address_start, address_end, value);
+ else
+ throw torrent::input_error("Invalid address format.");
return torrent::Object();
}
@@ -168,23 +138,176 @@ apply_ip_tables_add_address(const torrent::Object::list_type& args) {
// IPv4 filter functions:
//
+
+///////////////////////////////////////////////////////////
+// IPV4_RANGE_PARSE parses string into an ip range
+//
+// should be compatible with lines in p2p files
+// everything in address before colon is ignored
+//
+// ip range can be single ip in which case start=end
+// ip range can be cidr notation a.b.c.d/e
+// ip range can be explicit range like in p2p line a.b.c.d-w.x.y.z
+//
+// returns false if line does not contain valid ip or ip range
+// address_start and address_end will contain start and end ip
+//
+// addresses parsed are returned in host byte order
+// to get network byte order call htonl(address)
+///////////////////////////////////////////////////////////
+bool
+ipv4_range_parse(const char* address, uint32_t* address_start, uint32_t* address_end) {
+ char address_copy[4096]; // same length as buffer used to do reads so no worries about overflow
+ bool valid = false;
+ char address_start_str[20];
+ int address_start_index=0;
+ struct sockaddr_in sa_start;
+ *address_start=0;
+ *address_end=0;
+
+ // get rid of everything after '#' comments
+ // copy everything up to '#' to address_copy and work from there
+ while(address[address_start_index] != '#' && address[address_start_index] != '\r' &&
+ address[address_start_index] != '\n' && address[address_start_index] != '\0' &&
+ address_start_index < 4096 ) {
+ address_copy[address_start_index] = address[address_start_index];
+ address_start_index++;
+ }
+ address_copy[address_start_index] = '\0';
+ address_start_index=0;
+
+ // skip everything up to and including last ':' character and whitespace
+ const char* addr = strrchr(address_copy, ':');
+ addr = addr == NULL ? address_copy : addr+1;
+ while(addr[0] == ' ' || addr[0] == '\t')
+ addr++;
+
+ while( ((addr[0] >= '0' && addr[0] <= '9') || addr[0] == '.') && address_start_index < 19 ) {
+ address_start_str[address_start_index] = addr[0];
+ address_start_index++;
+ addr++;
+ }
+ address_start_str[address_start_index] = '\0';
+
+ if(strchr(addr, '-') != NULL) {
+ // explicit range
+ char address_end_str[20];
+ int address_end_index=0;
+ struct sockaddr_in sa_end;
+
+ while(addr[0] == '-' || addr[0] == ' ' || addr[0] == '\t')
+ addr++;
+
+ while( ((addr[0] >= '0' && addr[0] <= '9') || addr[0] == '.') && address_end_index < 19) {
+ address_end_str[address_end_index] = addr[0];
+ address_end_index++;
+ addr++;
+ }
+ address_end_str[address_end_index] = '\0';
+
+ if(inet_pton(AF_INET, address_start_str, &(sa_start.sin_addr)) != 0 && inet_pton(AF_INET, address_end_str, &(sa_end.sin_addr)) != 0) {
+ *address_start = ntohl(sa_start.sin_addr.s_addr);
+ *address_end = ntohl(sa_end.sin_addr.s_addr);
+ if(*address_start <= *address_end)
+ valid=true;
+ }
+ }
+ else if(strchr(addr, '/') != NULL) {
+ // cidr range
+ char mask_bits_str[20];
+ int mask_bits_index=0;
+ uint32_t mask_bits;
+ while(addr[0] == '/' || addr[0] == ' ' || addr[0] == '\t')
+ addr++;
+
+ while( (addr[0] >= '0' && addr[0] <= '9') && mask_bits_index < 19) {
+ mask_bits_str[mask_bits_index] = addr[0];
+ mask_bits_index++;
+ addr++;
+ }
+ mask_bits_str[mask_bits_index] = '\0';
+
+ if(inet_pton(AF_INET, address_start_str, &(sa_start.sin_addr)) != 0 && sscanf(mask_bits_str, "%u", &mask_bits) != 0) {
+ if(mask_bits <=32) {
+ uint32_t ip=ntohl(sa_start.sin_addr.s_addr);
+ uint32_t mask=0;
+ mask = (~mask) << (32-mask_bits);
+ *address_start = ip & mask;
+ uint32_t end_mask=0;
+ end_mask = (~end_mask) >> mask_bits;
+ *address_end = (ip & mask) | end_mask;
+ valid=true;
+ }
+ }
+ }
+ else {
+ // single ip
+ if(inet_pton(AF_INET, address_start_str, &(sa_start.sin_addr)) != 0) {
+ *address_start = ntohl(sa_start.sin_addr.s_addr);
+ *address_end = *address_start;
+ valid=true;
+ }
+ }
+ return valid;
+}
+
+
+
+
+
+///////////////////////////////////////////////////////////
+// IPV4_FILTER_PARSE
+//
+// should now be compatible with lines in p2p files
+//
+// addresses in table MUST be in host byte order
+// ntohl is called after parsing ip address(es)
+///////////////////////////////////////////////////////////
+void
+ipv4_filter_parse(const char* address, int value) {
+ uint32_t address_start;
+ uint32_t address_end;
+ if (ipv4_range_parse(address, &address_start, &address_end) ) {
+ torrent::PeerList::ipv4_filter()->insert(address_start, address_end, value);
+
+ char start_str[INET_ADDRSTRLEN];
+ char end_str[INET_ADDRSTRLEN];
+ uint32_t net_start = htonl(address_start);
+ uint32_t net_end = htonl(address_end);
+ inet_ntop(AF_INET, &net_start, start_str, INET_ADDRSTRLEN);
+ inet_ntop(AF_INET, &net_end, end_str, INET_ADDRSTRLEN);
+ lt_log_print(torrent::LOG_CONNECTION_DEBUG, "Adding ip filter for %s-%s.", start_str, end_str);
+ }
+}
+
+
+////////////////////////////////////////////////
+// APPLY_IPV4_FILTER_SIZE_DATA
+////////////////////////////////////////////////
torrent::Object
apply_ipv4_filter_size_data() {
return torrent::PeerList::ipv4_filter()->sizeof_data();
}
+
+////////////////////////////////////////////////
+// APPLY_IPV4_FILTER_GET
+////////////////////////////////////////////////
torrent::Object
apply_ipv4_filter_get(const std::string& args) {
- // Move to a helper function, add support for addresses.
- uint32_t ip_values[4];
-
- if (sscanf(args.c_str(), "%u.%u.%u.%u",
- ip_values + 0, ip_values + 1, ip_values + 2, ip_values + 3) != 4)
+ uint32_t address_start;
+ uint32_t address_end;
+ if (!ipv4_range_parse(args.c_str(), &address_start, &address_end))
throw torrent::input_error("Invalid address format.");
-
- return torrent::PeerList::ipv4_filter()->at((ip_values[0] << 24) + (ip_values[1] << 16) + (ip_values[2] << 8) + ip_values[3]);
+ if(!torrent::PeerList::ipv4_filter()->defined(address_start, address_end))
+ throw torrent::input_error("No value defined for specified IP(s).");
+ return torrent::PeerList::ipv4_filter()->at(address_start, address_end);
}
+
+////////////////////////////////////////////////
+// APPLY_IPV4_FILTER_ADD_ADDRESS
+////////////////////////////////////////////////
torrent::Object
apply_ipv4_filter_add_address(const torrent::Object::list_type& args) {
if (args.size() != 2)
@@ -195,6 +318,9 @@ apply_ipv4_filter_add_address(const torrent::Object::list_type& args) {
return torrent::Object();
}
+////////////////////////////////////////////////
+// APPLY_IPV4_FILTER_LOAD
+////////////////////////////////////////////////
torrent::Object
apply_ipv4_filter_load(const torrent::Object::list_type& args) {
if (args.size() != 2)
@@ -240,42 +366,36 @@ apply_ipv4_filter_load(const torrent::Object::list_type& args) {
return torrent::Object();
}
-static void
-append_table(torrent::ipv4_table::base_type* extent, torrent::Object::list_type& result) {
- torrent::ipv4_table::table_type::iterator first = extent->table.begin();
- torrent::ipv4_table::table_type::iterator last = extent->table.end();
-
- while (first != last) {
- if (first->first != NULL) {
- // Do something more here?...
- append_table(first->first, result);
-
- } else if (first->second != 0) {
- uint32_t position = extent->partition_pos(first);
-
- char buffer[256];
- snprintf(buffer, 256, "%u.%u.%u.%u/%u %s",
- (position >> 24) & 0xff,
- (position >> 16) & 0xff,
- (position >> 8) & 0xff,
- (position >> 0) & 0xff,
- extent->mask_bits,
- torrent::option_as_string(torrent::OPTION_IP_FILTER, first->second));
-
- result.push_back((std::string)buffer);
- }
-
- first++;
- }
-}
+////////////////////////////////////////////////
+// APPLY_IPV4_FILTER_DUMP
+////////////////////////////////////////////////
torrent::Object
apply_ipv4_filter_dump() {
torrent::Object raw_result = torrent::Object::create_list();
torrent::Object::list_type& result = raw_result.as_list();
+
- append_table(torrent::PeerList::ipv4_filter()->data(), result);
-
+ torrent::ipv4_table::range_map_type range_map = torrent::PeerList::ipv4_filter()->range_map;
+ torrent::ipv4_table::range_map_type::iterator iter = range_map.begin();
+ while(iter != range_map.end() ) {
+ char buffer[64];
+ uint32_t address_start = iter->first;
+ uint32_t address_end = (iter->second).first;
+ int value = (iter->second).second;
+
+ char start_str[INET_ADDRSTRLEN];
+ char end_str[INET_ADDRSTRLEN];
+ uint32_t net_start = htonl(address_start);
+ uint32_t net_end = htonl(address_end);
+ inet_ntop(AF_INET, &net_start, start_str, INET_ADDRSTRLEN);
+ inet_ntop(AF_INET, &net_end, end_str, INET_ADDRSTRLEN);
+
+ snprintf(buffer, 64, "%s-%s %s", start_str, end_str, torrent::option_as_string(torrent::OPTION_IP_FILTER, value));
+ result.push_back((std::string)buffer);
+
+ iter++;
+ }
return raw_result;
}
View
2  src/rpc/ip_table_list.h
@@ -45,7 +45,7 @@
namespace rpc {
-typedef torrent::extents<uint32_t, int, 32, 256, 8> ipv4_table;
+typedef torrent::extents<uint32_t, int> ipv4_table;
struct ip_table_node {
std::string name;
Something went wrong with that request. Please try again.