Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch backports to 5.0.x #6795

Merged
merged 17 commits into from
Jan 17, 2022
Merged

Batch backports to 5.0.x #6795

merged 17 commits into from
Jan 17, 2022

Conversation

jlucovsky
Copy link
Contributor

Continuation of #6738

Link to redmine ticket:

Describe changes:

Updates:

  • Address review comment: Removed Fedora 32 builds
  • Address review comment: Removed debug/informational commit from stream-tcp-reassemble.c [hash 55636c8]
    #suricata-verify-pr:
    #suricata-verify-repo:
    #suricata-verify-branch:
    #suricata-update-pr:
    #suricata-update-repo:
    #suricata-update-branch:
    #libhtp-pr:
    #libhtp-repo:
    #libhtp-branch:

Corey Thomas and others added 17 commits January 15, 2022 08:01
@jlucovsky
ci: add fedora 35 to builds
7a00c68 
(cherry picked from commit 7be793f)
@jasonish @jlucovsky
smtp: log transaction even if no email present
d3cafab 
The SMTP transaction logger was not writing the log if the email
portion of the logger failed, such as in the case of STARTTLS
where this is no email decoded.

Ticket OISF#4817

(cherry picked from commit 52b9c12)
@regit @jlucovsky
profiling: fix profiling with sample rate
ba14b58 
Rules profiling was returning invalid results when used with sample
rate. The problem was that the sample condition was run twice in the
packet flow. As a result, the second pass was not initializing the
variable storing the initial CPU ticks and the resulting performance
counters were reporting invalid values.

Bug: OISF#4836.
(cherry picked from commit 6d5f596)
@catenacyber @jlucovsky
conf: avoid quadratic complexity
64cc431 
Ticket: 4812

When adding many sequence nodes

(cherry picked from commit 1564942)
@jlucovsky
detect/file_data: Reject invalid protocols
215335f 
Issue: OISF#4895

This commit causes Suricata to reject signatures that combine TCP-based
alerts using file_data with NFS keywords.

file_data doesn't support the NFS protocol.
@jasonish @jlucovsky
dns: create transaction even if z-bit was set
63d8554 
It appears that DNS servers will still process a DNS request even if the
z-bit is set, our parser will fail the transaction. So create the
transaction, but still set the event.

Ticket OISF#4924

(cherry picked from commit fcbdc30)
@jlucovsky
dns: Logging of Z-bit
f9884ed 
[Edit by Jason Ish: fix flag bit value]

Ticket OISF#4515

(cherry picked from commit dfb6f10)
@jasonish @jlucovsky
dns: add dns flag to dns request logging
49ae710 
Ticket OISF#4515

(cherry picked from commit 0861b66)
@jlucovsky
doc/eve-json-format/dns: Describing Z-bit
f99934e 
(cherry picked from commit 2a5d79e)
@jlucovsky
suricatasc: Handle incomplete/empty recv values
6500683 
Issue: 4947

Improve handling of values returned by recv. Sometimes, recv returns an
empty string if suricata terminates asynchronously.

(cherry picked from commit fc6fdef)
@victorjulien @jlucovsky
stream: unify ack'd right edge handling
76c0507 
Use util function in all code needing the ack'd data.

(cherry picked from commit 258415b)
@victorjulien @jlucovsky
stream: fix stream pruning being too aggressive
e6ac610 
Pruning of StreamBufferBlocks could remove blocks that fell entirely
after the target offset due to a logic error. This could lead to data
being evicted that was still meant to be processed in theapp-layer
parsers.

Bug: OISF#4953.
(cherry picked from commit 78f5e08)
@jasonish @jlucovsky
af-packet: use configured cluster-id when checking for fanout
42045b9 
When testing for fanout support a cluster-id of 1 was always being
used instead of the configured cluster-id. This limited fanout
support to only one Suricata instance.

Instead of hardcoding an ID of 1, use the configured cluster-id.

Also make cluster_id a uint16_t instead of an int in AFPThreadVars.

Redmine issue:
https://redmine.openinfosecfoundation.org/issues/3419

(cherry picked from commit df0ed6f)
@jasonish @jlucovsky
datasets: initialize after dropping privileges
e1dd936 
Move initialization of datasets to a point after privileges
have been dropped.

Ticket 4239

(cherry picked from commit 92eb14c)
@victorjulien @jlucovsky
packetpool: reset PacketRelease on return to pool
68e16e2 
Reset PacketRelease callback to make sure its not set to a capture
specific callback.

As an example:

  0x000055e00af09d35 in AFPReleaseDataFromRing (p=0x7f1d884cb830) at source-af-packet.c:653
  0x000055e00af09dd0 in AFPReleasePacket (p=0x7f1d884cb830) at source-af-packet.c:678
  0x000055e00ab53d7e in TmqhOutputPacketpool (t=0x55e00fb79250, p=0x7f1d884cb830) at tmqh-packetpool.c:465
  0x000055e00af08dec in TmThreadsSlotProcessPkt (tv=0x55e00fb79250, s=0x55e012134790, p=0x7f1d884cb830) at tm-threads.h:201
  0x000055e00af08e70 in TmThreadsCaptureInjectPacket (tv=0x55e00fb79250, p=0x7f1d884cb830) at tm-threads.h:221
  0x000055e00af08f2e in TmThreadsCaptureHandleTimeout (tv=0x55e00fb79250, p=0x0) at tm-threads.h:245
  0x000055e00af0ba76 in ReceiveAFPLoop (tv=0x55e00fb79250, data=0x7f1d884ccb60, slot=0x55e01198e4b0) at source-af-packet.c:1321
  0x000055e00ab55257 in TmThreadsSlotPktAcqLoop (td=0x55e00fb79250) at tm-threads.c:312
  0x00007f1dca9d5609 in start_thread (arg=<optimized out>) at pthread_create.c:477
  0x00007f1dca7c6293 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:95

Here the packet was a pseudo packet to handle a timeout condition. But
the ReleasePacket callback was still set to AFPReleasePacket from a
previous use of the Packet.

Bug: OISF#4807.
(cherry picked from commit 07ce871)
@regit @jlucovsky
util/streaming: improve error handling
f663cdc 
It differentiates memory error than regular ones.

(cherry picked from commit 0e70958)
@regit @jlucovsky
stream: increase memcap on memory errors
3439aeb 
(cherry picked from commit c1bffa9)
@jlucovsky jlucovsky mentioned this pull request Jan 15, 2022
Closed
@suricata-qa
Copy link

WARNING:

ERROR: QA failed on tlpw1_files_sha256.

ERROR: QA failed on tlpw1_files_md5.

field test baseline %
tlpr1_stats_chk
.flow.memuse 582522880 663643184 87.78%

Pipeline 5709

@victorjulien victorjulien merged commit 3439aeb into OISF:master-5.0.x Jan 17, 2022
@jlucovsky jlucovsky deleted the batch-5.0.x/2 branch January 19, 2022 13:53
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 26, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
ff0757b 
Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 26, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
5cdef6e 
Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 27, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
02feaee 
Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 27, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
8b8d910 
Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 27, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
7d649c9 
Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 27, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
ee753a8 
Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 28, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
f2d113f 
Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Feb 28, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
39e85cf 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 1, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
071fb4b 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 1, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
9c58268 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 1, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
968b3e0 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 2, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
436257a 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 4, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
eaca03f 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 4, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
8434dab 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 4, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
863f28e 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
victorjulien added a commit to victorjulien/suricata that referenced this pull request Mar 4, 2024
@victorjulien
detect/port: use qsort instead of insert sort
e7e4305 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 5, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
03edb42 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 11, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
d7c2a6a 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 25, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
f9d67a3 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 28, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
e5ecb52 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
inashivb pushed a commit to inashivb/suricata that referenced this pull request Mar 29, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
77b2e27 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
inashivb pushed a commit to inashivb/suricata that referenced this pull request Apr 17, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
ffa1092 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
inashivb pushed a commit to inashivb/suricata that referenced this pull request Apr 18, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
18a871d 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
inashivb pushed a commit to inashivb/suricata that referenced this pull request May 6, 2024
@victorjulien @inashivb
detect/port: use qsort instead of insert sort
f13f2e0 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
victorjulien added a commit to victorjulien/suricata that referenced this pull request May 24, 2024
@victorjulien
detect/port: use qsort instead of insert sort
fb8c3c8 
Instead of using in place insertion sort on linked list based on two
keys, convert the linked list to an array, perform sorting on it using
qsort and convert it back to a linked list. This turns out to be much
faster.

Ticket OISF#6795

(cherry picked from commit e7e4305)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jasonish jasonish Awaiting requested review from jasonish jasonish is a code owner

@norg norg Awaiting requested review from norg norg is a code owner

@victorjulien victorjulien Awaiting requested review from victorjulien victorjulien is a code owner

Assignees
No one assigned
Labels
None yet
Milestone
No milestone
6 participants
@jlucovsky @suricata-qa @victorjulien @jasonish @regit @catenacyber