Use MessagePack for the Unix socket API #75

DL6ER · 2017-06-12T16:56:58Z

By submitting this pull request, I confirm the following (please check boxes, eg [X]) Failure to fill the template will close your PR:

Please submit all pull requests against the development branch. Failure to do so will delay or deny your request

I have read and understood the contributors guide.
I have checked that another pull request for this purpose does not exist.
I have considered, and confirmed that this submission will be valuable to others.
I accept that this submission may not be used, and the pull request closed at the will of the maintainer.
I give this submission freely, and claim no ownership to its content.

10

[Edited by Mcat12]

Change Unix socket interface to return data in MessagePack format. It still uses the same API commands as the telnet interface. The MessagePack format allows FTL to efficiently stream data in a very compact form to the upcoming HTTP API.

Also included are these smaller changes:

Limit the total number of client connections
Refactor checking the type of the connection into the global istelnet[] array of bools.
Improve the socket-test binary to allow custom commands to be sent.

This PR previously held an implementation of an HTTP API, but that will be moved to a separate program. It will access FTL's data through the Unix socket.

This template was created based on the work of udemy-dl.

DL6ER · 2017-06-12T16:58:38Z

Note that due to its complexity, this PR will most likely be a work in progress for quite some time and will not be part of the next versions of FTL we release

…ment removes the fixed buffer we used before and allows for responses having arbitraty lengths

…to decide whether the output whould be in telnet format, or JSON (possibly even "v2" JSON)

Currently using the cJSON library, as it's entirely contained within one source and one header file and is very easy to use. By adding on the API output formats to the previous raw socket code, we can simply change the output format when the response type is `API` instead of `SOCKET`. As shown in the implemented summary API code, this should be very simple for most commands.

…r into process_api_request() (request.c) much like we already do it for socket requests

…and http://pi.hole/admin/api.php?overTimeData10mins give identical results

…e by reverting this commit.

AzureMarker · 2017-06-14T00:24:44Z

request.c

-	if(sscanf(client_message, ">%*[^(](%i)", &num) > 0)
+	// SOCKET: >top-domains (15)
+	// API:    /stats/top_domains?limit=15
+	if(sscanf(client_message, "%*[^0123456789H\n]%i", &num) > 0)


Will this pick up /stats/top_domains?alkjsdlkfj=15 as valid?

AzureMarker · 2017-06-14T00:25:18Z

request.c

 	{
 		desc = true;
 	}
+	else if(type != SOCKET && command(client_message, "desc"))


This should check for order=desc

AzureMarker · 2017-06-14T00:29:12Z

request.c

+	// Match both top-domains and top-ads
+	// SOCKET: >top-clients (15)
+	// API:    /stats/top_clients?limit=15
+	if(sscanf(client_message, "%*[^0123456789H\n]%i", &num) > 0)


See other comment.

Also make query types output more like the other responses.

…the whole header was analyzed), 2. Anaylze passed GET arguments properly, e.g. allow ?limit=123, but ignore ?sfqw=123

Signed-off-by: Mcat12 <newtoncat12@yahoo.com> # Conflicts: # main.c # request.c # routines.h # socket.c

Removed reference to the HTTP API. Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Might have been lost in a merge. Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

DL6ER

Review comments for all files except api.c and request.c (need to use a better tool for looking at the diff, GitHub web doesn't seem to understand that the file got renamed).

DL6ER · 2018-01-20T20:08:07Z

FTL.h

@@ -23,6 +23,7 @@
 #include <time.h>
 #include <sys/time.h>
 #include <sys/socket.h>
+#include <regex.h>


Do we still use any subroutine from the regex.h header?

DL6ER · 2018-01-20T20:08:25Z

FTL.h

@@ -207,7 +211,8 @@ typedef struct {
 enum { DATABASE_WRITE_TIMER, EXIT_TIMER };

 enum { QUERIES, FORWARDED, CLIENTS, DOMAINS, OVERTIME, WILDCARD };
-enum { SOCKET };
+enum { TELNET, SOCKET };
+enum { WHITELIST, BLACKLIST, WILDLIST };


This might not be needed as well.

DL6ER · 2018-01-20T20:09:14Z

FTL.h

@@ -252,3 +257,5 @@ long int lastdbindex;
 bool travis;
 bool DBdeleteoldqueries;
 bool rereadgravity;
+char * clientip[MAXCONNS];


We might even want to get rid of all of this as we aren't implementing an authentication directly in FTL now.

DL6ER · 2018-01-20T20:10:48Z

main.c

 	pthread_t piholelogthread;
 	if(pthread_create( &piholelogthread, &attr, pihole_log_thread, NULL ) != 0)
 	{
 		logg("Unable to open Pi-hole log processing thread. Exiting...");
 		killed = 1;
 	}
+	sleepms(100);


I don't think this delay is needed at all.

DL6ER · 2018-01-20T20:11:05Z

main.c

 	pthread_t socket_listenthread;
 	if(pthread_create( &socket_listenthread, &attr, socket_listening_thread, NULL ) != 0)
 	{
 		logg("Unable to open Unix socket listening thread. Exiting...");
 		killed = 1;
 	}
+	sleepms(100);


This one can be removed as well, I think.

DL6ER · 2018-01-20T20:23:23Z

socket.c

-		return accept(sockfd, (struct sockaddr *) &in6_addr, &socklen);
+			memset(&in6_addr, 0, sizeof(in6_addr));
+			socklen = sizeof(un_addr);
+			char str[INET6_ADDRSTRLEN];


Can be removed

DL6ER · 2018-01-20T20:23:56Z

socket.c

@@ -293,7 +330,7 @@ void *telnet_connection_handler_thread(void *socket_desc)
 			// Requests should not be processed/answered when data is about to change
 			enable_thread_lock(threadname);

-			process_request(message, &sock);
+			process_request(message, &sock, TELNET);


TELNET is not needed here, as we have istelnet[sock]

DL6ER · 2018-01-20T20:24:20Z

socket.c

@@ -319,6 +356,11 @@ void *telnet_connection_handler_thread(void *socket_desc)
 		close(sock);
 	free(socket_desc);

+	if(clientip[sock] != NULL) {


Can be removed as well.

DL6ER · 2018-01-20T20:24:33Z

socket.c

@@ -351,7 +395,7 @@ void *socket_connection_handler_thread(void *socket_desc)
 			// Requests should not be processed/answered when data is about to change
 			enable_thread_lock(threadname);

-			process_request(message, &sock);
+			process_request(message, &sock, SOCKET);


Also here, SOCKET can be removed.

DL6ER · 2018-01-20T20:24:43Z

socket.c

@@ -376,6 +420,11 @@ void *socket_connection_handler_thread(void *socket_desc)
 		close(sock);
 	free(socket_desc);

+	if(clientip[sock] != NULL) {


Can remove this as well.

DL6ER · 2018-01-20T20:38:02Z

msgpack.c

+	// Make sure that the length is less than 4294967296
+	size_t length = strlen(string);
+
+	if(length >= 4294967296) {


msgpack.c: In function ‘pack_str32’: msgpack.c:95:12: warning: comparison is always false due to limited range of data type [-Wtype-limits] if(length >= 4294967296) {

When compiling on a 64 bit system, this warning does not appear. Do you think we should check for this?

4294967296 is slightly more than 4 GB, do we really expect strings to become that large?

If you want to keep the check (which is probably, a good idea), we can test for the length of size_t, maybe like this:

if((sizeof(size_t) > 4) && length >= 4294967296) { ... }

There shouldn't be any performance penalty here, as with any (non-zero) level of optimization the compiled binary won't actually perform the test as the compiler will already eliminate whichever branch is dead at compile time.

That code still gives the compiler warning. How about reducing it to:

if(length >= 2147483648) { ... }

The compiler does not yell about this, but it is a slightly sloppy way to avoid it.

I cannot test this at the moment but there should be a preprocessor constant for the maximum allowed size of size_t. Maybe something like

#if __SIZE_MAX__ > 4294967295 if(length > 4294967295) { ... } #endif

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER

There are some minor bugs and possible enhancements that need to be addressed in api.c but others than that it is nice work!

DL6ER · 2018-01-20T21:22:14Z

api.c

+	if(total > 0)
+		percentage = 1e2*blocked/total;
+
+	// MAX_INT is 10 digits, +1 for the null terminator


Why do we need these buffers at all? Instead of all the strncpy() things we can directly use ssend() (as it is done in development) to reduce code complexity and avoid unnecessary buffers.

You may want to look at https://github.com/pi-hole/FTL/blob/development/request.c#L245-L253 and https://github.com/pi-hole/FTL/blob/development/request.c#L274-L285 as I think that is nicely done here.

DL6ER · 2018-01-20T21:23:01Z

api.c

+			if(istelnet[*sock])
+				strncpy(domains_blocked, "N/A", 4);
+			else
+				strncpy(domains_blocked, "\"N/A\"", 6);


It doesn't seem that we need this at all (if sending over the Unix socket, we don't use this variable).

DL6ER · 2018-01-20T21:29:07Z

api.c

+{
+	int i, j = 9999999;
+
+	// Get first time slot with total or blocked greater than zero (the array will go down over time due to the rolling window)


"the array will go down over time due to the rolling window"

I had to read that twice to understand that you want to say that the individual entries will go to zero over time, maybe it should be clarified.

I think you wrote this comment ;)

DL6ER · 2018-01-20T21:32:37Z

api.c

+				if(istelnet[*sock])
+					ssend(*sock, "%i %i %s wildcard\n", n, domains[j].blockedcount, domains[j].domain);
+				else {
+					char *fancyWildcard = calloc(2 + strlen(domains[j].domain), sizeof(char));


You will have to allocate 3 + strlen as you have to store the string, *, . and the terminating NULL (strlen does not consider the NULL in the injected string when counting the length).

DL6ER · 2018-01-20T21:44:06Z

api.c

+	}
+	// Domain filtering?
+	if(command(client_message, ">getallqueries-domain")) {
+		sscanf(client_message, ">getallqueries-domain %ms", &domainname);


Just thinking out aloud: Could this %ms be a potential DDoS vector here? Like you send FTL some 100 MB through the telnet socket and fill up the memory? I think this was limited to some certain (fixed) length previously. I'm fine with %ms, but we should really check for the domainname pointer afterwards..

The Posix standard documents this extension in its POSIX.1-2008 edition:

The application shall be responsible for freeing the memory after usage. If there is insufficient memory to allocate a buffer, the function shall set errno to ENOMEM and a conversion error shall result.

The input is limited to 1024 chars, but they could just send the same command over and over again. Then it would only be limited by the max number of connections.

Here is how development currently handles it:
https://github.com/pi-hole/FTL/blob/master/request.c#L680-L685

This version was implemented in this commit:
48d355b

This version was implemented in this commit:
48d355b

Is this the nice way of blaming me for it? ;-) I'm not always and only having great ideas.

Do you want me to revert it back to the development way?

Yes, please. Maybe increase it to 256 if you find it necessary.

DL6ER · 2018-01-20T21:44:14Z

api.c

+	}
+	// Client filtering?
+	if(command(client_message, ">getallqueries-client")) {
+		sscanf(client_message, ">getallqueries-client %ms", &clientname);


See comment above

DL6ER · 2018-01-20T21:51:10Z

api.c

+	{
+		for(i = sendit; i < counters.overTime; i++)
+		{
+			double percentage;


We could go down to float as we truncate the digits anyhow later on. It would make the case on line api.c:902 unnecessary.

DL6ER · 2018-01-20T21:54:16Z

api.c

+		{
+			validate_access("overTime", i, true, __LINE__, __FUNCTION__, __FILE__);
+
+			double percentageIPv4 = 0.0, percentageIPv6 = 0.0;


Can use float here as well, removes casts in api.c:965f

DL6ER · 2018-01-20T21:57:04Z

api.c

+			sprintf(hashVersion, "vDev-%s", hash);
+
+			pack_str32(*sock, hashVersion);
+			pack_str32(*sock, (char *) tag);


Is this cast necessary? See also api.c:985.

I think constants (like GIT_HASH) are treated as const char* as they are pointers to a const location in space where this string is compiled into the binary.

There is a compiler warning otherwise:

api.c: In function 'getVersion': api.c:1007:22: warning: passing argument 2 of 'pack_str32' discards 'const' qualifier from pointer target type [-Wdiscarded-qualifiers] pack_str32(*sock, tag); ^~~ In file included from api.c:12:0: api.h:41:6: note: expected 'char *' but argument is of type 'const char *' void pack_str32(int sock, char *string); ^~~~~~~~~~

DL6ER · 2018-01-20T21:57:33Z

api.c

+		if(istelnet[*sock])
+			ssend(*sock, "version vDev-%s\ntag %s\nbranch %s\ndate %s\n", hash, tag, GIT_BRANCH, GIT_DATE);
+		else {
+			char *hashVersion = calloc(5 + strlen(hash), sizeof(char));


This has to be increased by one due to the terminating NULL character

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

# Conflicts: # FTL.h # socket.c

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

DL6ER · 2018-01-21T22:13:20Z

Approved 🎉

DL6ER added 2 commits June 12, 2017 18:52

Implement HTTP server into FTL, listening on port 4747 by default

941dfb2

Merge branch 'development' into new/API

38c706d

DL6ER and others added 17 commits June 12, 2017 19:11

Return error 404 if a browser is asking for favicon.ico

3b99d82

Interpret HEAD request

03fc5be

We don't allow caching the responds of FTL

e159b56

Replace sprintf() + swrite() by the new ssend() routine. This enhance…

19c54b0

…ment removes the fixed buffer we used before and allows for responses having arbitraty lengths

Add HTTP server tests

35a1882

Use silent mode if curl based tests to suppress progress meter display

a4fc933

Show last two lines of FTL logs when waiting for socket to become ready

d7d57a8

Add a small delay when starting the threads

f84b461

Test for both ports 4711 and 4747 before starting the tests

0ef2256

Add type variable to all request subroutines which can later be used …

b2688bf

…to decide whether the output whould be in telnet format, or JSON (possibly even "v2" JSON)

Only show API request HTTP data when debugging

7617293

Add Content-Length to 404 API response

5f59aff

Output JSON data w/o using any extra JSON module

e20fe2a

Clean socket routines, move detection of what the client is asking fo…

b15f537

…r into process_api_request() (request.c) much like we already do it for socket requests

Added overTime statistics to API, http://pi.hole:4747/stats/overTime …

1481176

…and http://pi.hole/admin/api.php?overTimeData10mins give identical results

Improve/add more comments to the code

824b9e9

DL6ER changed the base branch from master to development June 13, 2017 20:38

DL6ER added 3 commits June 13, 2017 23:53

Add /stats/top_domains and /stats/top_ads and parameters therein

4643f97

Remove cJSON files. Note that it can be re-added cleanly into the cod…

2b439cf

…e by reverting this commit.

Added /stats/top_clients

776bf43

AzureMarker reviewed Jun 14, 2017

View reviewed changes

AzureMarker and others added 5 commits June 13, 2017 21:37

Add /stats/forward_dest

7af31af

Add /stats/query_types

0e7f484

Add alias path for forward destinations

249b9e9

Also make query types output more like the other responses.

Make memory error in getQueryTypes not fatal

ea0a7cc

Two improvements: 1. Analyze only first line of HTTP request (before …

9cff885

…the whole header was analyzed), 2. Anaylze passed GET arguments properly, e.g. allow ?limit=123, but ignore ?sfqw=123

Merge remote-tracking branch 'origin/development' into new/API

d3d7098

Signed-off-by: Mcat12 <newtoncat12@yahoo.com> # Conflicts: # main.c # request.c # routines.h # socket.c

AzureMarker removed the Merge conflicts label Jan 14, 2018

AzureMarker added 4 commits January 20, 2018 11:11

Reword api.c license header description

cf0cdfe

Removed reference to the HTTP API. Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Reword api.h license description header

7c57b86

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Add back detailed ssend error message

1fd7ade

Might have been lost in a merge. Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Merge remote-tracking branch 'origin/development' into new/API

232507a

DL6ER commented Jan 20, 2018

View reviewed changes

DL6ER added 3 commits January 20, 2018 21:38

Implement some of the review comments

0aee819

Signed-off-by: DL6ER <dl6er@dl6er.de>

Remove clientips[]

df3b049

Signed-off-by: DL6ER <dl6er@dl6er.de>

Remove sleep from main()

996689f

Signed-off-by: DL6ER <dl6er@dl6er.de>

DL6ER commented Jan 20, 2018

View reviewed changes

AzureMarker added 2 commits January 20, 2018 21:40

Fix indentation in msgpack.c

9f5ae62

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Merge remote-tracking branch 'origin/development' into new/API

590f1f9

# Conflicts: # FTL.h # socket.c

AzureMarker changed the title ~~[WIP] Use MessagePack for the Unix socket API~~ Use MessagePack for the Unix socket API Jan 21, 2018

AzureMarker added 10 commits January 21, 2018 10:38

Don't use buffers in getStats

173efff

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Clarify getOverTime comment

76aff45

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Fix under-allocation in getTopDomains

ad99f97

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Use float instead of double in getForwardDestinationsOverTime

4e8027b

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Use float instead of double in getQueryTypesOverTime

be5815f

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Fix under-allocation in getVersion

9007af7

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Fix status output in getStats

42bf9a4

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Only allow up to 2GB strings to be sent

b6b7b57

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Use a static buffer for getallqueries string arguments

9bc0562

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

Handle possible pack_fixstr or pack_str32 error

5341b18

Signed-off-by: Mcat12 <newtoncat12@yahoo.com>

AzureMarker approved these changes Jan 21, 2018

View reviewed changes

AzureMarker merged commit fc0ef14 into development Jan 21, 2018

AzureMarker deleted the new/API branch January 21, 2018 22:45

DL6ER mentioned this pull request Jul 31, 2022

Major rewrite of the socket processing routines #1396

Merged

5 tasks

Use MessagePack for the Unix socket API #75

Use MessagePack for the Unix socket API #75

Conversation

DL6ER commented Jun 12, 2017 • edited by AzureMarker Loading

10

DL6ER commented Jun 12, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DL6ER left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DL6ER Jan 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DL6ER Jan 21, 2018 • edited Loading

Choose a reason for hiding this comment

DL6ER left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DL6ER commented Jan 21, 2018

DL6ER commented Jun 12, 2017 •

edited by AzureMarker

Loading

DL6ER Jan 21, 2018 •

edited

Loading

DL6ER Jan 21, 2018 •

edited

Loading