nDPI: use fuzzing script from repository by utoni · Pull Request #4041 · google/oss-fuzz

utoni · 2020-06-28T21:52:04Z

Signed-off-by: Toni Uhlig matzeton@googlemail.com

googlebot · 2020-06-28T21:52:08Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

utoni · 2020-06-28T21:55:21Z

@googlebot I signed it!

googlebot · 2020-06-28T21:55:24Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

* Changing the OSS-Fuzz from our side is much more easier then opening a PR to Google's oss-fuzz every time we have to change a single line. * google/oss-fuzz#4041 will be updated once this PR is merged Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

* Changing the OSS-Fuzz script from our side is much more easier then opening a PR to google/oss-fuzz every time we have to change a single line. * google/oss-fuzz#4041 will be updated once this PR is merged Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

Adrian Zgorzałek (2): OpenBSD: Introduce pkt_timeval to deal with (bpf_)_timeval OpenBSD: Do not redefine __LITTLE_ENDIAN__ Alexander Czyrny (2): Fix for missing line breaks in CSV file additional csv semicolon fix Alfredo Cardigliano (23): Moved to 3.3 Check NULL strings in ndpi_serialize_string_string Enable IEC 60870-5-104 dissector Handle empty blocks Add ndpi_serialize_start_of_list/ndpi_serialize_end_of_list to serialize simple lists in JSON Add ndpi_serializer_get_header API (CSV only) Support for multiple records in CSV serialization Handle EOR in TLV test Add API ndpi_serializer_get_format Fix segfault on ndpi_guess_protocol_id with flow = null Fix warnings Guide update Add start_of_block/end_of_block support to TLV Add missing low-level serializer calls to the API Replace snprintf with ndpi_snappend Temporarily keep using snprintf Add unit tests to travis. Move ndpi serializer tests to unit tests. Add distdir directive Exclude sanitizer on unit tests involving json-c due to a bug in the lib Serialized doxygen doc Decoupled fuzzy and unit tests Add serialization of values list in TLV Handle list items in ndpi_deserialize_get_single_size Bas Alberts (1): ql query to identify suspicious use of network sourced integers Disconnect3d (2): Fix off by one when checking for "GET / HTTP" string Fix incorrect "<iq from=\"' parsing Ivan Nardi (1): QUIC: fix dissection of "offset" field (#1025) Leonn Paiva (3): 🆒 code ident & style 🐛 use protocol_was_guessed in ndpi_detection_giveup 💡 implement websocket protocol dissector Luca (2): Commented unused function Added ndpi_quick_16_byte_hash Luca Deri (241): Fixed swapped protocol Compilation fix Compilation fix Added SMBv1 over NetBIOS detection NetBIOS dissection improvements Win fixes Update for QUIC 046 Added QUIC v046 test pcap Fixed loop in serialization Added dos_win98_smb_netbeui.pcap Added netbios.pcap Added dns_exfiltration.pcap Added badpackets.pcap Added android.pcap Added iphone.pcap Added smb_deletefile.pcap Added netflow-fritz.pcap Added wa_voice.pcap Added wa_video.pcap Added fuzz-2006-06-26-2594.pcap fuzzy pcap Added fuzz-2006-09-29-2858 fuzzy pcap Added fuzz-2020-02-16-11 fuzzy pcap Format update Format update Fixed invalid allocation NDPI_API_VERSION is now computed automatically Compilation fix Added s7comm test pcap Fix for string serialization with initial empty string Minor refresh Fixes #853 and adds a self check for duplicates Added ndpi_serialize_raw_record() API call Invalid function definition fix Restored ndpi_get_api_version() prototype Fixed invalid check Added detection of Microsoft Teams Results update CiscoVPN dissection improvements Added check for pkg-config presence Compilation fixes Updated node list (fixes #766) Office365 renamed to Microsoft365 (by Microsoft) Updated results Fix for nDPI code not downlooaded from git Warning fix Minor cleanup SSH boundary check rework Removed now obsolete MSN protocol Reworked protocol handling chnging it is u_int16_t Updated API False positive fixes Various fixes to patricia tree handling Introduced custom protocols with IP and (optional) port support Added support for Telegram v6 Updated automa API to use 32 bit values splits from protocol/categpry API cleanup for indetifying explicitly in automa's what we're searching (protocol or category) Fixed category matching Reworked TLS dissector with a certificate RDN sequence reader Added TLS issuerDN and subjectDN Added self signed certificate test pcap Cleaned hyperscan leftover Added the ability to detect when a known protocol is using a non-standard port Added detection of self-signed TLS certificates Added TLS weak cipher and obsolete protocol version detection Added NDPI_XXX_BIT_16 Invalid TLS check Code cleanup Code clenup for PR #902 Improvements on GotoMeeting Gotomeeting address range fix Added link to nfstream Added NDPI_TLS_CERTIFICATE_EXPIRED, NDPI_TLS_CERTIFICATE_MISMATCH, to ndpi_risk Added NDPI_HTTP_SUSPICIOUS_USER_AGENT ndpi_risk Fix for potential heap-buffer-overflow in ndpi_search_openvpn Added check for binary scripts Added check for invalid HTTP URLs Added fix to avoid potential heap buffer overflow in H.323 dissector Removed bittorrent false positive detection Fixed valse positive whatsapp detection MS Teams uses as underlying protocol for voice/video. This commit adds the ability Added stub for checking HTTP header Added memory boundary checks Added flow risk serilization Added ndpi_serialize_risk() to the nDPI API Extended the cache for services that need to be reconciled such as Microsoft Teams Added pcap with encrypted SNI Result update Added ndpi_serialize_risk() API function Added support for Encrypted TLS SNI dissection Refreshed test pcap Added references to ESNI Fixes for ntop/nDPI#911 Added check for heap buffer overflow read Minor cleanup HTTP dissector improvements Added check to avoid producing alerts for known protocol on unknown port when using TLS Added extra TLS memory boundary checks Added offset check in kerberos dissector Reworked ndpi_strncasestr Minor changes HTTP comparison optimization Reworked HTTP header processing workflow Check to avoid allocating twice the esni memory Added check for spotify payload lenght Added boundary check in kerberos protocol Improved spotify detection Improved teamviewer detection Added ndpi_dpi2json() API call removed obsolete yahoo plugin Removed some obsolete protocols (battlefield, oscar, pcanywhere, tvants) Added TLS bounadry check Added HTTP check Added check in TLS 1.2+ for reporting a risk when TLS is not used to carry HTTPS Win fixes Updated (C) Added HyperLogLog cardinality estimator API calls Added DGA risk for names that look like a DGA Added improved boundary check Fixed invalid assignment (typo) Added checks for DGA detection DGA detection improvements Fixed API documentation: packet tiestamp is expressed in milliseconds Added GoogleDNS DoH on Android 10 Added ndpi_bin_XXX API Warning fix Fixes #906 Disabled unused code Values stored in patricia tree are now 32 bit (they used to be 16 bit) long Added malformed packet risk support Added notes whenever a new flow risk is added Use clang where available and fallback to gcc when missing Restored compiler check for fuzzy targets Added ndpi_bin_similarity() for computing bin similarity Added ndpi_print_bin() API call Indent Make clean enhancement Added unit test Added comment on stddev Added ndpi_cluster_bins() for clustering bins and ancillary functions for bins manipulation Memory allocation fixes Added testing bin code Added -b to ndpiReader to evaluate bins Various fixes in bins implementation Updated test results due to bin changes wrapper cleanup Fixed makefile error message Check for avoiding buffer overflow in netbios dissector Add ndpi_hll_reset() API call Added ndpi_data_window_variance() and ndpi_data_window_stddev() API calls Added ndpi_data_last() API call Indendentation fix Added ndpi_reset_data_analysis() API call Fix for invalid boundary check Added skeleton for checking SSH signature User agent detection improvements HLL memory usage notes Introduced SSH rick checks Minor HLL fixes Added changes for handlign SSSH cipher detection Improved bin clustering Fixed bin similarity SSH code cleanup Added parentheses to avoid issues with macro expansion Boundary check Boundary check on QUIC Added NDPI_SMB_INSECURE_VERSION for detecting insecure SMB versions (e.g. v1) TLS dissection improvements Tiny changes for TLS block lenght dissection Restored TLS dissection Changed due to bin size extension Fixed partial TLS dissection Added memory checks Added new traffic category for connectivity check detection Added check on payload lenght during extra packet processing Win #define fix Fixed possible memory leak in TLS certificate handling Added new ndpi_string_sha1_hash API call Added note on memory management Added case-insensitive substring matching Added HLL notes Improved DGA detection algoritm Fixes invalid detection on traffic on non standard ports Added support for discord Added README for building libndpi under OpenWRT Added --with-only-libndpi configure option to build just lindpi Added fix for API versions starting with 0 Updated OpenWRT instructions Compilation fix Configure code cleanup MySQL8 update Added the ability do identigy as DGA those host/domain names with too many consucutive repeated characters Added new check for detecting suspicious (too long) names Warning fix Fixes compilation issues introduced by ntop/nDPI#989 Added som GQUIC and IETF QUIC test pcaps Added libgcrypt20-dev dependency to handle QUIC Added QUIC dependency Cosmetic fix Creared IoT-Scada category Added ndpi_http_method2str() API call Added ndpi_http_method ndpi_http_str2method(const char* method) API call Fixed false positive in suspicous user agent Stddev calculation changes Improved ntop detection over HTTP Added new risk for NDPI_UNSAFE_PROTOCOL that identifies protocols that are not condidered safe/secure Added (optional) notifier for LRU add Added check for ndpi_ssl_version2str() Added trademark information Added boundary check Added some additional TLS mappings Added command for specifying the aux files dir Added extension to detect nested subdomains as used in Browsertunnel attack tool Added buffer check in DGA analysis Added boundary check Reworked DGA checks Disabled QUIC tracing that pollutes the output Reworked MDNS dissector that is not based on the DNS dissector Added MDNS in extra packet dissection function Updated results Added sample Windows MSVC project Added risks for checking Minor change for alignment issue Minor UA handling improvement to avoid heap-overflow Macro redefinition Improved boundary check to prevent overflow Attempt to avoid errors due to misaligned packet headers -O0 -> -Og Added -fno-sanitize=alignment in clang-7 Various optimizations to reduce not-necessary calls Compilation fix Boundary fix Added back ndpi_check_flow_func (correct) prototype Added extra boundary checks Added JSON-C check for unit tests Updated serialization test unit Added fix for invalid SNI check when SNI is missing Tests update Warning fix Improved processing of IPv6 header Added makefile target for creating changelog 3.4 nDPI Release MrRadix (13): added ssh obsolete version risks added ssh_analyse_signature_version and ssh_has_old_signature for check old signature version of ssh improved performance by removing linear scan added new risks iside ndpi_risk2str function added new risks to ndpi_risk_enum improved ndpi_risk2str output for new risks added modified risks improved performance and legibility added sscanf error handling modified new last two risks fixed bug inside set bit macro call added cipher check added other ssh implementations to check Nardi Ivan (51): ciscovpn: fix heap-overflow error soulseek: fix heap-overflow error h323: fix heap-overflow error quic: fix heap-overflow error ssh: fix heap-overflow error tls: fix heap-overflow error kerberos: fix use-after-free error telnet: fix heap-overflow error telnet: fix another heap-overflow error openvpn: fix heap-overflow error Fix an harmless memory leak s7comm: fix heap-overflow error kerberos: fix heap-overflow error Fix some debug messages ndpiReader: fix memory leak in idle sessions purging Fix some compilation warnings Add basic support for some ip-in-ip tunnels Fix "division by zero" runtime error Fix heap-overflow error in CAPWAP detunneling code Fix NATS dissector Restore extcap functionality, i.e. integration with wireshark Fixed harmless memory leak in extcap initialization Make lua script more robust Fix compilation with --enable-debug-messages flag mqtt: add boundary check in log message TLS: extract JA3 signatures in some corner cases ndpiReader: fix ports statistics Fix startup when DPDK is enabled Fix (harmless) memory leaks when DPDK is enabled DNP3: add missing initialization Fix use-after-free in http content parsing Fix memory leak about purged/expired flows Fix undefined behaviour in internal tests Update test results Incorporated some feedback Improve help message of --dbg-proto option Fix a memory leak Fix memory leak reported in #955 Minor fixes Add risk flag about suspicious ESNI usage Suspicious ESNI usage: add a comment and a pcap example Major rework of QUIC dissector Add (optional) dependency on external libraries: libgcrypt and libgpg-error Update TLS dissector to handle QUIC flows Add sub-classification for GQUIC >= Q050 and (IETF-)QUIC QUIC: minor fixes QUIC: add support for GQUIC T050 and T051 http: create a common function to parse User Agent field QUIC: extract User Agent information TLS: fix memory accesses in QUIC transport parameters extension QUIC: add support for MVFST EXPERIMENTAL version Nguyen Phuong An (1): ndpi_flow2json should check http.url before serializer Philippe Antoine (34): Fix read overflow before UDP header Fix off by one in imap Run ndpi_detection_process_packet only with payload Adds check before memcpy for bittorrent Fix buffer overread in ndpi_search_setup_capwap Fix snprintf return value check for tls Fix buffer over read in dns Fix buffer overread in netbios Fix buffer overread in yahoo Fix infinite loop in tls Checks enough data for UDP header Adds different checks against overflows Fix use of uninitialized value in TLS tls_supported_versions only if version_str is initialized ssh: adds systematic bounds checks in concat_hash_string ssh: fixing unsigned overflow leading to heap overflow Fixes OOB reads in postgres Adds bound check in TLS Fix integer overflow in quic Adds bound check for TZSP Adds bound check for IRC Use ndpi_handle_ipv6_extension_headers in reader_util Adds tls check before reading memory TLS initializes version_str Adds netbios bound check Adds bound check before calling ndpi_handle_ipv6_extension_headers Adds CIFuzz to check PRs Seeting right flow protocol after IP6 extensions Gets right protocol after IPv6 header Better fix for integer overflow in SSH Adds bound check in oracle protocol Enables fuzz targets even without sanitizer Reenables CI with sanitizers msan cxx Simone Mainardi (4): Fixed API change which removed ndpi_netbios_name_interpret Fixes netbios overread Passes method_len param to ndpi_http_str2method Fixes control reaches end of non-void function Tommaso Macchioni (1): Added check for suspicious HTTP headers Toni (2): Added missing files to `make dist' target which are not required to build nDPI but still somehow essential. (#1024) Fixed missing PCAP include directories in Makefiles. (#1034) Toni Uhlig (60): Added missing ndpiReader dependency for the install target Fixed docstring typos for ndpi_finalize_initalization Implemented proprietary AnyDesk protocol Fixed stack overflow caused by missing length check make tests/do.sh directory agnostic Fixed use after free caused by dangling pointer Allow -fsanitize for LLVM as well as for GCC. Fixed unitialized values in ndpiReader protocol detection bitmask during dga selftest. Added hulustream SNI pattern. Fixes #904. Fixed missing length check in fbzero. ndpiSimpleIntegration: added another integration example fixed autoconf version check fixed fuzzing tests in a way that ./tests/do.sh is now able to use corpus *.pcap files from ./tests/pcap Fixed off-by-one error in h323. Re-Added corpus zipfile generation required for google-oss fuzzer Improved fuzz_ndpi_reader which supports now SMP/MT w/o race-coniditions. Copy&Paste ./tests/ossfuzz.sh from google/oss-fuzz#4041 Added fuzz_*_LDADD to give Automake the opportunity to do some dependency tracking. Fixed heap overflow in tls esni extraction triggered by manipulated packets. Improved TeamViewer IP range (fixes #944) added shell script to download wireshark fuzzing traces, can be used in combination with ./tests/do.sh Improved HTTP line parsing if request splitted into multiple packets. ossfuzz.sh: do not use wildcards for fuzzer e.g. fuzz/fuzz* Fixed CodeInspector issues. fix broken sanitizer CFLAGS, --enable-fuzztargets should also set -fsanitize Fixed heap overflow caused by missing lengthcheck in reader uutil. fixed memory leak in ndpi_cluster_bins / binUnitTest Fixed thunder protocol detection heap overflow caused by missing lengthcheck. Removed csv_fp as external symbol. Instead passing csv_fp through as argument. ndpiSimpleIntegration should not be included in a static library travis-ci cleanups Fixed race condition in ndpi_ssl_version2str() caused by static qualifier in the version string buffer. Fixed heap overflow caused by missing lengthcheck in 802.11 LLC header parsing. fixes autoconf issue on ubuntu 18 (#953) Changed ndpi_ssl_version2str function call in ndpiSimpleIntegration. travis-ci: build ndpiSimpleIntegration as well num_extra_packets_checked check can be 0 for some protocols and therefor requires lesser-or-equal condition for max_extra_packets_to_check Replaced obsolete libpcap pcap_lookupdev with pcap_findalldevs. Fixed invalid dpdk fn call. Added support for SOAP. Updated MySQL protocol detection to support server version 8. Added (manipulated) MySQL 8 test pcap. Fixed broken pkg-config file which did not care about gcrypt/pcre. Fixed broken travis-ci YAML. Moved NDPI_CURRENT_PROTO define before ndpi_api.h include to prevent a redefinition warning. Fixed use-of-uninitialized-value in QUIC clho decryption probably caused by a BUG in libgcrypt (not verified). Fixed travis-ci fuzzm job. Might be a false positive related to clang-7's MSAN. Fixed invalid memory access leading to a SIGSEGV in ndpiReader's option parser. Fixed false positive detection for Skype.SkypeCall (affects at least Cisco HSRP and RADIUS). Fixed off-by-one error in Kerberos protocol. Improved dnscrypt v1/v2 protocol detection. Updated DoT/DoH provider. Added dnscrypt-v2-doh resolver test pcaps. Added pcap file which contains dnscrypt-v1 data and resolver update requests/responses (v1/v2). Fixed shlib xcompile for x86_64-w64-mingw32 Improved Teamspeak(3) protocol detection. Fixed mingw build w/o examples/tests/fuzzer. Using NDPI_I64_FORMAT, NDPI_U64_FORMAT format string to differentiate between Windows and non-Windows. Fixed broken `make dist' and added CI check. Do not re-define libc functions for mingw builds. Zied Aouini (5): Add ndpi_flow_tcp_struct and ndpi_flow_udp_struct api calls. Fix docstrings. Remove trailing alignment directives. Update Python cffi bindings. Fix flow_printer example. aouinizied (12): Update APIs and structures. Synchronize type definitions. Add NDPI_MALFORMED_PACKET risk. Update note. Add ssl_version_str to flow structure. Add note for sync with cffi bindings. Update CFFI definitions. Sync TLS definitions. Fix typo. Add Connectivity check category and blacklisted host risk. Update example. Fix CAPWAP handling. borisVanhoof (1): don't pass arguments to ./configure as 1 string (#1027) emanuele-f (2): Fix ndpi_get_api_version version truncation Fix API version generation havup (11): Update ndpi_protocol_ids.h Update ndpi_protocols.h Update ndpi_main.c Update ndpi_main.c Add files via upload Update s7comm.c Update s7comm.c Update tls_long_cert.pcap.out Update s7comm.c Update s7comm.c Update s7comm.c loures (4): Add check for HTTP transfer of executable files Set risk field instead of changing protocol when checking for dangerous Extend packet struct with Content-Disposition HTTP header field Extend filetype matching for Content-Disposition header lucaderi (6): CentOS6 fix (santize won't work as too old system) Compilation fixes for non-Linux (or outdated Linux) platforms Check for CentOS 6 configure fixes Complation fix as not on all platforms isnumber() is available Added missing install target in newly added unit tsts ysk (7): someip#fix when run across architectures such ARM ,may be occour sigbus error(),Unhandled fault: alignment fault (0x92000021) at 0x00000000f67004aa nDpi#remove build warnings fix segment fault cause by the ssl.server_names when it may NULL fix segment fault cause by the ssl.server_names when it may NULL add improved boundary check and check malloc return is NULL 1:add free pointer NULL check;2:fix xbox and teredo protocol detected error when use the commm udp port 3544 remove the unused code

utoni mentioned this pull request Jun 29, 2020

Copy&Paste ./tests/ossfuzz.sh from https://github.com/google/oss-fuzz… ntop/nDPI#941

Merged

utoni changed the title ~~nDPI: make use of a new Makefile target to generate the corpus~~ nDPI: use fuzzing script from repository Jul 1, 2020

nDPI: use fuzzing script from repository

8390dc7

Signed-off-by: Toni Uhlig <matzeton@googlemail.com>

utoni mentioned this pull request Jul 5, 2020

ossfuzz.sh: do not use wildcards for fuzzer e.g. fuzz/fuzz* ntop/nDPI#951

Merged

utoni closed this Jul 6, 2020

utoni mentioned this pull request Jul 6, 2020

excludes builds artifacts from ndpi project #4077

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

nDPI: use fuzzing script from repository#4041

nDPI: use fuzzing script from repository#4041
utoni wants to merge 1 commit intogoogle:masterfrom
utoni:master

utoni commented Jun 28, 2020

Uh oh!

googlebot commented Jun 28, 2020

Uh oh!

utoni commented Jun 28, 2020

Uh oh!

googlebot commented Jun 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

utoni commented Jun 28, 2020

Uh oh!

googlebot commented Jun 28, 2020

What to do if you already signed the CLA

Individual signers

Corporate signers

Uh oh!

utoni commented Jun 28, 2020

Uh oh!

googlebot commented Jun 28, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants