Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use MessagePack for the Unix socket API #75

Merged
merged 183 commits into from
Jan 21, 2018
Merged
Show file tree
Hide file tree
Changes from 168 commits
Commits
Show all changes
183 commits
Select commit Hold shift + click to select a range
941dfb2
Implement HTTP server into FTL, listening on port 4747 by default
DL6ER Jun 12, 2017
38c706d
Merge branch 'development' into new/API
DL6ER Jun 12, 2017
3b99d82
Return error 404 if a browser is asking for favicon.ico
DL6ER Jun 12, 2017
03fc5be
Interpret HEAD request
DL6ER Jun 12, 2017
e159b56
We don't allow caching the responds of FTL
DL6ER Jun 12, 2017
19c54b0
Replace sprintf() + swrite() by the new ssend() routine. This enhance…
DL6ER Jun 12, 2017
35a1882
Add HTTP server tests
DL6ER Jun 12, 2017
a4fc933
Use silent mode if curl based tests to suppress progress meter display
DL6ER Jun 12, 2017
d7d57a8
Show last two lines of FTL logs when waiting for socket to become ready
DL6ER Jun 12, 2017
f84b461
Add a small delay when starting the threads
DL6ER Jun 12, 2017
0ef2256
Test for both ports 4711 and 4747 before starting the tests
DL6ER Jun 12, 2017
b2688bf
Add type variable to all request subroutines which can later be used …
DL6ER Jun 12, 2017
1d29630
Implement API JSON output for summary
AzureMarker Jun 13, 2017
7617293
Only show API request HTTP data when debugging
AzureMarker Jun 13, 2017
5f59aff
Add Content-Length to 404 API response
AzureMarker Jun 13, 2017
e20fe2a
Output JSON data w/o using any extra JSON module
DL6ER Jun 13, 2017
b15f537
Clean socket routines, move detection of what the client is asking fo…
DL6ER Jun 13, 2017
1481176
Added overTime statistics to API, http://pi.hole:4747/stats/overTime …
DL6ER Jun 13, 2017
824b9e9
Improve/add more comments to the code
DL6ER Jun 13, 2017
4643f97
Add /stats/top_domains and /stats/top_ads and parameters therein
DL6ER Jun 13, 2017
2b439cf
Remove cJSON files. Note that it can be re-added cleanly into the cod…
DL6ER Jun 13, 2017
776bf43
Added /stats/top_clients
DL6ER Jun 13, 2017
7af31af
Add `/stats/forward_dest`
AzureMarker Jun 14, 2017
0e7f484
Add `/stats/query_types`
AzureMarker Jun 14, 2017
249b9e9
Add alias path for forward destinations
AzureMarker Jun 14, 2017
ea0a7cc
Make memory error in getQueryTypes not fatal
AzureMarker Jun 14, 2017
9cff885
Two improvements: 1. Analyze only first line of HTTP request (before …
DL6ER Jun 14, 2017
f2f9601
Greatly simplify code by not sending Content-Length (optional accordi…
DL6ER Jun 14, 2017
48d355b
Add "/history"
DL6ER Jun 14, 2017
b5c500f
Fix all queries time filtering and SOCKET output determination
DL6ER Jun 14, 2017
2179f5d
Don't put integer values in quotes
DL6ER Jun 14, 2017
b868c56
Add "/dns/recent_blocked" (can take argument ?limit=123)
DL6ER Jun 14, 2017
2aa2ff1
Added "/overTime/forward_dest", renamed "/stats/overTime" -> "/overTi…
DL6ER Jun 14, 2017
9d891d5
Added "/overTime/query_types"
DL6ER Jun 14, 2017
70e3b5d
Move "/overTime/*" -> "/stats/overTime/*"
DL6ER Jun 14, 2017
38ea0a7
Use thread locks for API responses
DL6ER Jun 14, 2017
942250e
Also send "dns_queries_today" in Top Clients request
DL6ER Jun 14, 2017
30a3f7e
Also send "dns_queries_today" in Top Domains / Ads requests
DL6ER Jun 14, 2017
59fe873
Return what is aksed for (top_domains)
DL6ER Jun 14, 2017
5df3233
Send two additional counter values along with the top lists
DL6ER Jun 14, 2017
ecc2ef3
Do some formatting
AzureMarker Jun 15, 2017
446a1b9
Add GET /dns/whitelist and /dns/blacklist
AzureMarker Jun 15, 2017
cce43dc
It is no severe issue if the lists (e.g. blacklist) are not there - s…
DL6ER Jun 15, 2017
1905f9c
Return data with key "blacklist" or "whitelist" instead of "data". Us…
DL6ER Jun 15, 2017
0e3f8b5
Use getline() instead of the deprecated fgets() to avoid havng to use…
DL6ER Jun 15, 2017
7cef472
Imporve triming of newline character when reading in the lists.
DL6ER Jun 15, 2017
364ad81
Merge branch 'development' into new/API
DL6ER Jun 15, 2017
c8f349d
Stop at "&" after domain/client filtering strings
DL6ER Jun 15, 2017
cb9d1cd
Add "/dns/status"
DL6ER Jun 15, 2017
a06ecba
Make sure to send API headers even if there's no data
AzureMarker Jun 16, 2017
693e69b
Send API headers every time & simplify getList
AzureMarker Jun 16, 2017
6cab551
Merge branch 'development' into new/API
DL6ER Jun 16, 2017
c29ce3b
Add first API test
DL6ER Jun 16, 2017
3f330dd
Make silent
DL6ER Jun 17, 2017
f8072a0
Minor correction to API tests
DL6ER Jun 17, 2017
ec47c81
Adjusted tests
DL6ER Jun 17, 2017
17a7823
Merge branch 'development' into new/API
DL6ER Jun 18, 2017
9893ef6
Adjust tests with new black.list interpretation
DL6ER Jun 18, 2017
5df0395
Change key name of forwards over time JSON
AzureMarker Jun 26, 2017
0dfb855
Merge branch 'development' into new/API
DL6ER Jun 26, 2017
f566fd3
Fix tests
DL6ER Jun 26, 2017
bbb3df2
Add a missing space, our tests require 100% accuracy
DL6ER Jun 26, 2017
cfea5cf
Only give relevant total in top domain JSON output
AzureMarker Jun 27, 2017
542e26c
Merge remote-tracking branch 'origin/new/API' into new/API
AzureMarker Jun 27, 2017
5693b72
Fix top domains and ads tests
AzureMarker Jun 27, 2017
e0f462e
The newline?
AzureMarker Jun 27, 2017
08391f4
Rename /stats/overTime/graphs to graph
AzureMarker Jun 30, 2017
52d3364
Rename /history to /stats/history
AzureMarker Jun 30, 2017
7c88707
Rename /dns/recent_blocked to /stats/recent_blocked
AzureMarker Jun 30, 2017
ccbf0ee
Add "GET /stats/dashboard"
DL6ER Jul 4, 2017
32c2a41
Don't send header multiple times
DL6ER Jul 4, 2017
c9de13b
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Jul 18, 2017
0a6f5a5
Sanitize domains when outputting `/stats/history`
AzureMarker Jul 29, 2017
345ac16
Revert "Sanitize domains when outputting `/stats/history`"
AzureMarker Jul 29, 2017
4762b8a
Reorganize to split up API functions
AzureMarker Aug 6, 2017
f64251e
Merge branch 'development' into new/API
DL6ER Aug 7, 2017
551351b
We don't show "pi.hole" anymore in the results. Adjust tests (also fo…
DL6ER Aug 7, 2017
ebe0631
Add dummy POST handling for whitelist and blacklist
AzureMarker Aug 9, 2017
9b98101
Fix JSON "not_found" message
AzureMarker Aug 9, 2017
9ec7bd8
Implement simple JSON parsing for addList
AzureMarker Aug 10, 2017
2f8e540
Check if domain is invalid
AzureMarker Aug 10, 2017
3a5752d
Actually add the domain to the list
AzureMarker Aug 10, 2017
a743931
Improve API response method to use enums
AzureMarker Aug 11, 2017
87dc82c
Add extra character for null terminator
AzureMarker Aug 11, 2017
1893aa5
Add whitelist and blacklist deletion API endpoints
AzureMarker Aug 11, 2017
06be1c4
Move debug API request logging to see all requests
AzureMarker Aug 12, 2017
729d5d4
Respond to CORS preflight (OPTIONS)
AzureMarker Aug 12, 2017
fb574f7
Perform better request endpoint checking
AzureMarker Aug 17, 2017
4842c16
Fix broken domain verification if the domain includes numbers
AzureMarker Sep 4, 2017
834a959
Add Wildlist support
AzureMarker Sep 4, 2017
e5c4d19
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Sep 26, 2017
e2f2f0d
Fix tests
AzureMarker Sep 27, 2017
3331c42
Move getPayload to api.c
AzureMarker Nov 27, 2017
604ad7e
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Dec 23, 2017
3573f54
Fix getVersion after merge
AzureMarker Dec 23, 2017
894bd39
Fix newly merged in API functions
AzureMarker Dec 23, 2017
1f5ca04
Add status to test summary data
AzureMarker Dec 23, 2017
4e90a6e
Add quotation marks around new summary status
AzureMarker Dec 23, 2017
2212e23
Fix domains_being_blocked output
AzureMarker Dec 23, 2017
5005f28
Implement simple password-protection to the HTTP API
AzureMarker Dec 24, 2017
ae81270
Fix invalid for loop error
AzureMarker Dec 24, 2017
cd2fd11
Make client IPs globally available in FTL (array of strings clientip[…
DL6ER Dec 25, 2017
f15b9ed
Make sure the API handler ignores clients when over the limit
AzureMarker Dec 25, 2017
636183b
Validate domain earlier in removeList
AzureMarker Dec 25, 2017
c9e40a4
Free client IP data after API handler runs and close rejected sockets
AzureMarker Dec 26, 2017
6050aa5
Add IP address into the authentication check
AzureMarker Dec 26, 2017
8adb3d3
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Jan 3, 2018
f62f9f0
Disable HTTP tests as authorization is not gained in the tests yet
AzureMarker Jan 3, 2018
3d78cc3
Fix tests
AzureMarker Jan 3, 2018
595ab25
Remove HTTP API specific code so it can be replaced with a new protocol
AzureMarker Jan 4, 2018
620bb44
Remove duplicate dnsmasq config file location
AzureMarker Jan 4, 2018
97f6ac5
Remove extra parameter from bind_to_telnet_port
AzureMarker Jan 4, 2018
c0048e9
Remove Json libraries
AzureMarker Jan 4, 2018
e527985
Implement getStats using MessagePack
AzureMarker Jan 4, 2018
439227a
Don't declare variables in for loops
AzureMarker Jan 4, 2018
b5c9c6e
Fix socket test
AzureMarker Jan 4, 2018
616b10a
Add swrite function
AzureMarker Jan 4, 2018
203cc94
Add istelnet array
DL6ER Jan 4, 2018
f0793be
Use istelnet[] in getStats()
DL6ER Jan 4, 2018
e54b391
Add socket implementation for getOverTime
AzureMarker Jan 6, 2018
c0022a9
Use istelnet for getOverTime
AzureMarker Jan 6, 2018
adfcacc
Use istelnet for getAllQueries
AzureMarker Jan 6, 2018
c6369dd
Remove old HTTP argument parsing from getAllQueries
AzureMarker Jan 6, 2018
4eaf794
Tighten parameters on pack_unsigned_char
AzureMarker Jan 6, 2018
4386138
Use uint8_t instead of unsigned char in msgpack.c
AzureMarker Jan 6, 2018
348a2ee
Add socket implementation for getAllQueries
AzureMarker Jan 6, 2018
7d3dfb6
Don't send fixarray for getAllQueries
AzureMarker Jan 6, 2018
e07786e
Remove dead code in getAllQueries
AzureMarker Jan 7, 2018
05f8604
Add socket implementation to getTopDomains and use istelnet
AzureMarker Jan 7, 2018
00f8877
Add socket implementation to getTopClients and use istelnet
AzureMarker Jan 7, 2018
33dd350
Add socket implementation to getForwardDestinations and use istelnet
AzureMarker Jan 7, 2018
3f1ffaf
Add socket implementation to getQueryTypes and use istelnet
AzureMarker Jan 7, 2018
555a0cb
Add socket implementation to getRecentBlocked and use istelnet
AzureMarker Jan 7, 2018
55e18ea
Remove extra formatting changes
AzureMarker Jan 7, 2018
81200a2
Add socket implementation to getMemoryUsage and use istelnet
AzureMarker Jan 8, 2018
89ea996
Fix pack_uint64
AzureMarker Jan 8, 2018
7d0f680
Add socket implementation to getForwardDestinationsOverTime and istelnet
AzureMarker Jan 8, 2018
0814339
Add socket implementation to getClientID and use istelnet
AzureMarker Jan 8, 2018
779a824
Add socket implementation to getQueryTypesOverTime and use istelnet
AzureMarker Jan 8, 2018
bd63981
Remove the unused `first` boolean variables (used by Json serialization)
AzureMarker Jan 8, 2018
2671a6f
Add socket implementation of getVersion and use istelnet
AzureMarker Jan 8, 2018
64a1408
Add socket implementation of getDBstats and use istelnet
AzureMarker Jan 8, 2018
f90f6c8
Use ssend in getClientsOverTime
AzureMarker Jan 8, 2018
dde1602
Add socket implementation for getClientsOverTime and use istelnet
AzureMarker Jan 8, 2018
1482742
Use ssend in getClientNames
AzureMarker Jan 8, 2018
43716b4
Add socket implementation to getClientNames and use istelnet
AzureMarker Jan 8, 2018
da7c354
Add socket implementation to getUnknownQueries and use istelnet
AzureMarker Jan 8, 2018
30ae391
Let socket-test take in an optional command
AzureMarker Jan 8, 2018
4ed20a4
Don't show a warning if write() returns 0 (this isn't an error)
AzureMarker Jan 9, 2018
37360bb
Send forward destinations names and IP addresses alongside overTime data
AzureMarker Jan 9, 2018
f02c4e8
Reimplement output for getList
AzureMarker Jan 11, 2018
b9780dc
Reimplement output for getPiholeStatus
AzureMarker Jan 11, 2018
7d87722
Remove getList
AzureMarker Jan 11, 2018
3158f17
Remove getPiholeStatus
AzureMarker Jan 12, 2018
cf2aff7
Remove addList
AzureMarker Jan 12, 2018
071f924
Remove the rest of the DNS commands and unused helpers
AzureMarker Jan 12, 2018
1983d3d
Rename api_stats.c to api.c
AzureMarker Jan 12, 2018
700a21b
Update the Makefile with new API file structure
AzureMarker Jan 12, 2018
353e0cb
Move command() up to the top of the file so it isn't as hidden
AzureMarker Jan 12, 2018
0013edb
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Jan 12, 2018
31c422e
Fix checkClientLimit
AzureMarker Jan 12, 2018
9b8b232
Fix incorrect merging
AzureMarker Jan 12, 2018
c88af99
Add `*.` to domains in getTopDomains if it's for audit and a wildcard
AzureMarker Jan 14, 2018
d3d7098
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Jan 14, 2018
cf0cdfe
Reword api.c license header description
AzureMarker Jan 20, 2018
7c57b86
Reword api.h license description header
AzureMarker Jan 20, 2018
1fd7ade
Add back detailed ssend error message
AzureMarker Jan 20, 2018
232507a
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Jan 20, 2018
0aee819
Implement some of the review comments
DL6ER Jan 20, 2018
df3b049
Remove clientips[]
DL6ER Jan 20, 2018
996689f
Remove sleep from main()
DL6ER Jan 20, 2018
9f5ae62
Fix indentation in msgpack.c
AzureMarker Jan 21, 2018
590f1f9
Merge remote-tracking branch 'origin/development' into new/API
AzureMarker Jan 21, 2018
173efff
Don't use buffers in getStats
AzureMarker Jan 21, 2018
76aff45
Clarify getOverTime comment
AzureMarker Jan 21, 2018
ad99f97
Fix under-allocation in getTopDomains
AzureMarker Jan 21, 2018
4e8027b
Use float instead of double in getForwardDestinationsOverTime
AzureMarker Jan 21, 2018
be5815f
Use float instead of double in getQueryTypesOverTime
AzureMarker Jan 21, 2018
9007af7
Fix under-allocation in getVersion
AzureMarker Jan 21, 2018
42bf9a4
Fix status output in getStats
AzureMarker Jan 21, 2018
b6b7b57
Only allow up to 2GB strings to be sent
AzureMarker Jan 21, 2018
9bc0562
Use a static buffer for getallqueries string arguments
AzureMarker Jan 21, 2018
5341b18
Handle possible pack_fixstr or pack_str32 error
AzureMarker Jan 21, 2018
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 8 additions & 1 deletion FTL.h
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
#include <time.h>
#include <sys/time.h>
#include <sys/socket.h>
#include <regex.h>
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we still use any subroutine from the regex.h header?

// struct sockaddr_in
#include <netinet/in.h>
// char* inet_ntoa(struct in_addr in)
Expand Down Expand Up @@ -72,6 +73,9 @@
// Default -60 (one minute before a full hour)
#define GCdelay (-60)

// How many client connection do we accept at once?
#define MAXCONNS 20

// Static structs
typedef struct {
const char* conf;
Expand Down Expand Up @@ -207,7 +211,8 @@ typedef struct {
enum { DATABASE_WRITE_TIMER, EXIT_TIMER };

enum { QUERIES, FORWARDED, CLIENTS, DOMAINS, OVERTIME, WILDCARD };
enum { SOCKET };
enum { TELNET, SOCKET };
enum { WHITELIST, BLACKLIST, WILDLIST };
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might not be needed as well.

enum { DNSSEC_UNSPECIFIED, DNSSEC_SECURE, DNSSEC_INSECURE, DNSSEC_BOGUS, DNSSEC_ABANDONED, DNSSEC_UNKNOWN };

logFileNamesStruct files;
Expand Down Expand Up @@ -252,3 +257,5 @@ long int lastdbindex;
bool travis;
bool DBdeleteoldqueries;
bool rereadgravity;
char * clientip[MAXCONNS];
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We might even want to get rid of all of this as we aren't implementing an authentication directly in FTL now.

bool istelnet[MAXCONNS];
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@
# This file is copyright under the latest version of the EUPL.
# Please see LICENSE file for your rights under this license.

DEPS = FTL.h routines.h version.h
OBJ = main.o structs.o log.o daemon.o parser.o signals.o socket.o request.o grep.o setupVars.o args.o flush.o threads.o gc.o config.o database.o
DEPS = FTL.h routines.h api.h version.h
OBJ = main.o structs.o log.o daemon.o parser.o signals.o socket.o request.o grep.o setupVars.o args.o flush.o threads.o gc.o config.o database.o api.o msgpack.o

# Get git commit version and date
GIT_BRANCH := $(shell git branch | sed -n 's/^\* //p')
Expand Down
Loading