Skip to content
Permalink
Browse files

implemented timeouts for sources

  • Loading branch information
Homas committed Mar 22, 2020
2 parents b39e007 + c1bb015 commit e97e771fb2361daae255da12e0b0cd77e1a5daa0
Showing with 17 additions and 14 deletions.
  1. +3 −0 ChangeLog.md
  2. +10 −10 README.md
  3. +1 −1 TODO.md
  4. +2 −2 include/ioc2rpz.hrl
  5. +1 −1 src/ioc2rpz_conn.erl
  6. BIN wiki_images/walled_garden.png
@@ -1,5 +1,8 @@
# ioc2rpz change log
[CB] - Changed Behaviour
## 2020-02-20 v1.1.1.4
- [CB] A default timeout (SourcePullTimeout - 5 minutes) was added to limit time for fetching feeds/sources via http/https/ftp.

## 2019-12-11 v1.1.1.3
- [CB] IoC lookup REST API call. The submitted indicator converted to lowcase before the lookups.

@@ -374,18 +374,18 @@ kdig @94.130.30.123 -y hmac-sha256:ioc2rpz-YOUR-UNIQUE-KEY-NAME:ioc2rpz-YOUR-UNI
You can find other IOC feeds on the wiki-page: https://github.com/Homas/ioc2rpz/wiki/IOC-Sources.
## References
- [Domain Name System (DNS) IANA Considerations](https://tools.ietf.org/html/rfc6895)
- [Domain Names - Implementation and Specification](https://tools.ietf.org/html/rfc1035)
- [Incremental Zone Transfer in DNS](https://tools.ietf.org/html/rfc1995)
- [RFC-6895 Domain Name System (DNS) IANA Considerations](https://tools.ietf.org/html/rfc6895)
- [RFC-1035 Domain Names - Implementation and Specification](https://tools.ietf.org/html/rfc1035)
- [RFC-1995 Incremental Zone Transfer in DNS](https://tools.ietf.org/html/rfc1995)
- [DNS Response Policy Zones (RPZ)](https://tools.ietf.org/html/draft-ietf-dnsop-dns-rpz-00) + [vixie](https://tools.ietf.org/html/draft-vixie-dns-rpz-02)
- [Secret Key Transaction Authentication for DNS (TSIG)](https://tools.ietf.org/html/rfc2845)
- [HMAC: Keyed-Hashing for Message Authentication](https://tools.ietf.org/html/rfc2104)
- [HMAC SHA TSIG Algorithm Identifiers](https://tools.ietf.org/html/rfc4635)
- [DNS Transport over TCP - Implementation Requirements](https://tools.ietf.org/html/rfc5966)
- [A Mechanism for Prompt Notification of Zone Changes (DNS NOTIFY)](https://tools.ietf.org/html/rfc1996)
- [RFC-2845 Secret Key Transaction Authentication for DNS (TSIG)](https://tools.ietf.org/html/rfc2845)
- [RFC-2104 HMAC: Keyed-Hashing for Message Authentication](https://tools.ietf.org/html/rfc2104)
- [RFC-4635 HMAC SHA TSIG Algorithm Identifiers](https://tools.ietf.org/html/rfc4635)
- [RFC-5966 DNS Transport over TCP - Implementation Requirements](https://tools.ietf.org/html/rfc5966)
- [RFC-1996 A Mechanism for Prompt Notification of Zone Changes (DNS NOTIFY)](https://tools.ietf.org/html/rfc1996)
- [Extension Mechanisms for DNS (EDNS(0))](https://tools.ietf.org/html/rfc6891) + [ENDS Option Codes](https://www.iana.org/assignments/dns-parameters/dns-parameters.xhtml#dns-parameters-11)
- [Domain Name System (DNS) Cookies](https://tools.ietf.org/html/rfc7873)
- [Specification for DNS over Transport Layer Security (TLS)](https://tools.ietf.org/html/rfc7858)
- [RFC-7873 Domain Name System (DNS) Cookies](https://tools.ietf.org/html/rfc7873)
- [RFC-7858 Specification for DNS over Transport Layer Security (TLS)](https://tools.ietf.org/html/rfc7858)
- [Cowboy Web Server](https://ninenines.eu)
- [Rebar3](https://www.rebar3.org)
@@ -1,5 +1,5 @@
## Bugs
- [x] wildcard rule is generated
- [ ] Take a look on the bugs mentioned in REST section

## Core / DNS
- [x] check IoC in the RPZs feeds "What's in your DNS?"
@@ -43,12 +43,12 @@
-define(HotCacheTime,900). %900 Time to cache IOCs/Records/Pkts in the hot cache. More usefull for online rpz.
-define(HotCacheTimeIXFR,0). %Time to cache IXFR IOCs in a hot cache. By default it is cached for a minute because of curr_serial_60.
-define(ShellMaxRespSize,2*1024*1024*1024). %Maximum response size for shell source

-define(SourcePullTimeout, 5 * 60 * 1000). %Default source/feed download timeout. Fetching will be interrupted if this download time exceeds it.

%%%%%%
%%%%%% Do not modify any settings below the line
%%%%%%
-define(ioc2rpz_ver, "1.1.1.3-2019127001").
-define(ioc2rpz_ver, "1.1.1.4-2020022001").

-define(ZNameZip,16#c00c:16). %Zone name/original fqdn from a request is always at byte 10 in the response
-define(ZNameZipN,16#c00c). % Offset in bytes - Zone name/original fqdn from a request is always at byte 10 in the response
@@ -81,7 +81,7 @@ get_ioc(<<"shell:",CMD/binary>> = _URL, Retry) ->
%download IOCs from http/https/ftp
get_ioc(<<Proto:5/bytes,_/binary>> = URL, Retry) when Proto == <<"http:">>;Proto == <<"https">>;Proto == <<"ftp:/">> ->
httpc:set_options([{cookies,enabled}]),
case httpc:request(get,{binary_to_list(URL),[{"User-Agent", "Mozilla"}]},[],[{body_format,binary},{sync,true}]) of %,{socket_opts,[{cookies,enabled}]}
case httpc:request(get,{binary_to_list(URL),[{"User-Agent", "Mozilla"}]},[{timeout, ?SourcePullTimeout}],[{body_format,binary},{sync,true}]) of %,{socket_opts,[{cookies,enabled}]}
{ok,{{_,200,_},_,Response}} ->
{ok,Response};
{ok,{{_,Code,_},Headers,Response}} ->
Binary file not shown.

0 comments on commit e97e771

Please sign in to comment.
You can’t perform that action at this time.