ERROR: can't set --dump-dynamic-rules /tmp/tha_rules/so_rules/ and no rules are being imported.

[felbinger]

Hello,
I tried to install your application according to the snort 3 Installation guide for Ubuntu (page 9 - 11).
When I try to execute the last command one page 10 (sudo /usr/local/bin/pulledpork.pl -c /usr/local/etc/pulledpork/pulledpork.conf -l -P -E -H SIGHUP), I get an error and no rules are being imported

I guess this is the interesting part, but you can find the whole output and the configuration below:

Generating Stub Rules....
    Generating shared object stubs via:/usr/local/bin/snort -c /usr/local/etc/snort/snort.conf --dump-dynamic-rules=/tmp/tha_rules/so_rules/
    An error occurred: ERROR: can't set --dump-dynamic-rules /tmp/tha_rules/so_rules/

    An error occurred: ERROR: usage: --dump-dynamic-rules output stub rules for all loaded rules libraries

    An error occurred: FATAL: see prior 2 errors

    An error occurred: Fatal Error, Quitting..

    Done
    Reading rules...
    Reading rules...

Verbose Output (-vvv) of the mentioned command

    https://github.com/shirkdog/pulledpork
      _____ ____
     `----,\    )
      `--==\\  /    PulledPork v0.8.0 - The only positive thing to come out of 2020...well this and take-out liquor!
       `--==\\/
     .-~~~~-.Y|\\_  Copyright (C) 2009-2021 JJ Cummings, Michael Shirk
  @_/        /  66\_  and the PulledPork Team!
    |    \   \   _(")
     \   /-| ||'--'  Rules give me wings!
      \_\  \_\\
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
Config File Variable Debug /usr/local/etc/pulledpork/pulledpork.conf
    sid_changelog = /var/log/sid_changes.log
    block_list = /usr/local/etc/lists/default.blocklist
    temp_path = /tmp
    distro = FreeBSD-12
    snort_path = /usr/local/bin/snort
    ignore = deleted.rules,experimental.rules,local.rules
    IPRVersion = /usr/local/etc/lists
    rule_url = ARRAY(0x55bb38e47dd0)
    rule_path = /usr/local/etc/rules/snort.rules
    snort_control = /usr/local/bin/snort_control
    sid_msg = /usr/local/etc/snort/sid-msg.map
    pid_path = /var/log/snort/snort.pid
    sorule_path = /usr/local/etc/so_rules/
    sid_msg_version = 2
    config_path = /usr/local/etc/snort/snort.conf
    local_rules = /usr/local/etc/rules/local.rules
    version = 0.8.0
    ips_policy = security
MISC (CLI and Autovar) Variable Debug:
    Process flag specified!
    arch Def is: x86-64
    Operating System is: linux
    CA Certificate File is: OS Default
    Config Path is: /usr/local/etc/pulledpork/pulledpork.conf
    Distro Def is: FreeBSD-12
    Write ONLY enabled rules flag is Set
    security policy specified
    local.rules path is: /usr/local/etc/rules/local.rules
    Rules file is: /usr/local/etc/rules/snort.rules
    sid changes will be logged to: /var/log/sid_changes.log
    sid-msg.map Output Path is: /usr/local/etc/snort/sid-msg.map
    Sending signal Flag is Set: SIGHUP
    Snort Version is: 3.1.0.0
    Snort Config File: /usr/local/etc/snort/snort.conf
    Snort Path is: /usr/local/bin/snort
    SO Output Path is: /usr/local/etc/so_rules/
    Will process SO rules
    Logging Flag is Set
    Verbose Flag is Set
    File(s) to ignore = deleted.rules,experimental.rules,local.rules
    Base URL is: https://www.snort.org/rules/|snortrules-snapshot.tar.gz|MY_OINKCODE https://snort.org/downloads/ip-block-list|IPBLOCKLIST|open
    doh, we need to perform some cleanup ... an unclean run last time?
Cleanup....
    removed 2 temporary snort files or directories from /tmp/tha_rules!
Checking latest MD5 for snortrules-snapshot-3100.tar.gz....
    Fetching md5sum for: snortrules-snapshot-3100.tar.gz.md5
** GET https://www.snort.org/rules/snortrules-snapshot-3100.tar.gz.md5?oinkcode=MY_OINKCODE ==> 200 OK
    most recent rules file digest: 89f05dbaa731ff94434bd60c1d02e49f
    current local rules file  digest: 89f05dbaa731ff94434bd60c1d02e49f
    The MD5 for snortrules-snapshot-3100.tar.gz matched 89f05dbaa731ff94434bd60c1d02e49f

IP Blocklist download of https://snort.org/downloads/ip-block-list....
** GET https://snort.org/downloads/ip-block-list ==> 302 Found
** GET https://snort-org-site.s3.amazonaws.com/production/document_files/files/000/004/600/original/ip_filter.blf?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIAIXACIED2SPMSC7GA%2F20210215%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Date=20210215T163750Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=dc6a7df1349a66743cb3202b0ddf0ebb313b5279966c84a799bb6f85c5486668 ==> 200 OK (1s)
    Reading IP List...
Prepping rules from snortrules-snapshot-3100.tar.gz for work....
    extracting contents of /tmp/snortrules-snapshot-3100.tar.gz...
    Ignoring plaintext rules: deleted.rules
    Ignoring plaintext rules: experimental.rules
    Ignoring plaintext rules: local.rules
    Reading rules...
Snort 3.0 detected, future Snort 3.0 processing
Generating Stub Rules....
    Generating shared object stubs via:/usr/local/bin/snort -c /usr/local/etc/snort/snort.conf --dump-dynamic-rules=/tmp/tha_rules/so_rules/
    An error occurred: ERROR: can't set --dump-dynamic-rules /tmp/tha_rules/so_rules/

    An error occurred: ERROR: usage: --dump-dynamic-rules output stub rules for all loaded rules libraries

    An error occurred: FATAL: see prior 2 errors

    An error occurred: Fatal Error, Quitting..

    Done
    Reading rules...
    Reading rules...
Cleanup....
    removed 2 temporary snort files or directories from /tmp/tha_rules!
Writing Blocklist File /usr/local/etc/lists/default.blocklist....
Writing Blocklist Version 858940980 to /usr/local/etc/lists/IPRVersion.dat....
Activating security rulesets....
    Done
Setting Flowbit State....
    Done
Writing /usr/local/etc/rules/snort.rules....
    Done
Generating sid-msg.map....
    Done
Writing v2 /usr/local/etc/snort/sid-msg.map....
    Done
WARNING, cannot send signal if also processing SO rules
    see README.SHAREDOBJECTS
    or use -T flag!
Writing /var/log/sid_changes.log....
    Done
Rule Stats...
    New:-------0
    Deleted:---0
    Enabled Rules:----0
    Dropped Rules:----0
    Disabled Rules:---0
    Total Rules:------0
IP Blocklist Stats...
    Total IPs:-----789

Done
Please review /var/log/sid_changes.log for additional details
Fly Piggy Fly!

/usr/local/etc/pulledpork/pulledpork.conf

# Config file for pulledpork
# Be sure to read through the entire configuration file
# If you specify any of these items on the command line, it WILL take 
# precedence over any value that you specify in this file!

#######
#######  The below section defines what your oinkcode is (required for 
#######  VRT rules), defines a temp path (must be writable) and also 
#######  defines what version of rules that you are getting (for your 
#######  snort version and subscription etc...)
####### 

# You can specify one or as many rule_urls as you like, they 
# must appear as http://what.site.com/|rulesfile.tar.gz|1234567.  You can specify
# each on an individual line, or you can specify them in a , separated list
# i.e. rule_url=http://x.y.z/|a.tar.gz|123,http://z.y.z/|b.tar.gz|456
# note that the url, rule file, and oinkcode itself are separated by a pipe |
# i.e. url|tarball|123456789, 
rule_url=https://www.snort.org/rules/|snortrules-snapshot.tar.gz|MY_OINKCODE
# NEW Community ruleset:
#rule_url=https://snort.org/downloads/community/|community-rules.tar.gz|Community
# NEW For IP Block lists! Note the format is urltofile|IPBLOCKLIST|<oinkcode>
# This format MUST be followed to let pulledpork know that this is a blocklist
rule_url=https://snort.org/downloads/ip-block-list|IPBLOCKLIST|open
# THE FOLLOWING URL is for emergingthreats downloads, note the tarball name change!
# and open-nogpl, to avoid conflicts.
#rule_url=https://rules.emergingthreats.net/|emerging.rules.tar.gz|open-nogpl
# THE FOLLOWING URL is for etpro downloads, note the tarball name change!
# and the et oinkcode requirement!
#rule_url=https://rules.emergingthreatspro.com/|etpro.rules.tar.gz|<et oinkcode>
# NOTE above that the VRT snortrules-snapshot does not contain the version
# portion of the tarball name, this is because PP now automatically populates
# this value for you, if, however you put the version information in, PP will
# NOT populate this value but will use your value!

# Specify rule categories to ignore from the tarball in a comma separated list
# with no spaces.  There are four ways to do this:
# 1) Specify the category name with no suffix at all to ignore the category
#    regardless of what rule-type it is, ie: netbios
# 2) Specify the category name with a '.rules' suffix to ignore only gid 1
#    rulefiles located in the /rules directory of the tarball, ie: policy.rules
# 3) Specify the category name with a '.preproc' suffix to ignore only
#    preprocessor rules located in the /preproc_rules directory of the tarball,
#    ie: sensitive-data.preproc
# 4) Specify the category name with a '.so' suffix to ignore only shared-object
#    rules located in the /so_rules directory of the tarball, ie: netbios.so
# The example below ignores dos rules wherever they may appear, sensitive-
# data preprocessor rules, p2p so-rules (while including gid 1 p2p rules),
# and netbios gid-1 rules (while including netbios so-rules):
# ignore = dos,sensitive-data.preproc,p2p.so,netbios.rules
# These defaults are reasonable for the VRT ruleset with Snort 2.9.0.x.
ignore=deleted.rules,experimental.rules,local.rules
# IMPORTANT, if you are NOT yet using 2.8.6 then you MUST comment out the
# previous ignore line and uncomment the following!
# ignore=deleted,experimental,local,decoder,preprocessor,sensitive-data

# What is our temp path, be sure this path has a bit of space for rule 
# extraction and manipulation, no trailing slash
temp_path=/tmp

#######
#######  The below section is for rule processing.  This section is 
#######  required if you are not specifying the configuration using
#######  runtime switches.  Note that runtime switches do SUPERSEED 
#######  any values that you have specified here!
#######

# What path you want the .rules file containing all of the processed 
# rules? (this value has changed as of 0.4.0, previously we copied 
# all of the rules, now we are creating a single large rules file
# but still keeping a separate file for your so_rules!
rule_path=/usr/local/etc/rules/snort.rules

# What path you want the .rules files to be written to, this is UNIQUE
# from the rule_path and cannot be used in conjunction, this is to be used with the
# -k runtime flag, this can be set at runtime using the -K flag or specified
# here.  If specified here, the -k option must also be passed at runtime, however
# specifying -K <path> at runtime forces the -k option to also be set
# out_path=/usr/local/etc/snort/rules/

# If you are running any rules in your local.rules file, we need to
# know about them to properly build a sid-msg.map that will contain your
# local.rules metadata (msg) information.  You can specify other rules
# files that are local to your system here by adding a comma and more paths...
# remember that the FULL path must be specified for EACH value.
# local_rules=/path/to/these.rules,/path/to/those.rules
local_rules=/usr/local/etc/rules/local.rules

# Where should I put the sid-msg.map file?
sid_msg=/usr/local/etc/snort/sid-msg.map

# New for by2 and more advanced msg mapping.  Valid options are 1 or 2
# specify version 2 if you are running barnyard2.2+.  Otherwise use 1
sid_msg_version=2

# Where do you want me to put the sid changelog?  This is a changelog 
# that pulledpork maintains of all new sids that are imported
sid_changelog=/var/log/sid_changes.log
# this value is optional

#######
#######  The below section is for so_rule processing only.  If you don't
#######  need to use them.. then comment this section out!
#######  Alternately, if you are not using pulledpork to process 
#######  so_rules, you can specify -T at runtime to bypass this altogether
#######

# What path you want the .so files to actually go to *i.e. where is it
# defined in your snort.conf, needs a trailing slash
sorule_path=/usr/local/etc/so_rules/

# Path to the snort binary, we need this to generate the stub files
snort_path=/usr/local/bin/snort

# We need to know where your snort.conf file lives so that we can
# generate the stub files
config_path=/usr/local/etc/snort/snort.conf

##### Deprecated - The stubs are now  categorically written to the  single rule file!
# sostub_path=/usr/local/etc/snort/rules/so_rules.rules

# Define your distro, this is for the precompiled shared object libs!
# Valid Distro Types:
# Alpine-3-10
# Centos-6, Centos-7, Centos-8
# Debian-8, Debian-9, Debian-10
# FC-27, FC-30
# FreeBSD-11, FreeBSD-12
# OpenBSD-6-2, OpenBSD-6-4, OpenBSD-6-5,
# OpenSUSE-15-0, OpenSUS-15-1, OpenSUSE-42-3
# RHEL-6, RHEL-7, RHEL-8
# Slackware-14-2
# Ubuntu-14-4, Ubuntu-16-4, Ubuntu-17-10, Ubuntu-18-4
distro=FreeBSD-12

#######  This next section is optional, but probably pretty useful to you.
#######  Please read thoroughly!

# If you are using IP Reputation and getting some public lists, you will probably
# want to tell pulledpork where your blocklist file lives, PP automagically will
# de-dupe any duplicate IPs from different sources.
block_list=/usr/local/etc/lists/default.blocklist

# IP Reputation does NOT require a full snort HUP, it introduces a concept whereby
# the IP list can be reloaded while snort is running through the use of a control
# socket.  Please be sure that you built snort with the following optins:
# -enable-shared-rep and --enable-control-socket.  Be sure to read about how to
# configure these!  The following option tells pulledpork where to place the version
# file for use with control socket ip list reloads!
# This should be the same path where your block_list lives!
IPRVersion=/usr/local/etc/lists

# The following option tells snort where the snort_control tool is located.
snort_control=/usr/local/bin/snort_control

# What do you want to backup and archive?  This is a comma separated list
# of file or directory values.  If a directory is specified, PP will recurse
# through said directory and all subdirectories to archive all files.
# The following example backs up all snort config files, rules, pulledpork
# config files, and snort shared object binary rules.
# backup=/usr/local/etc/snort,/usr/local/etc/pulledpork,/usr/local/lib/snort_dynamicrules/

# what path and filename should we use for the backup tarball?
# note that an epoch time value and the .tgz extension is automatically added
# to the backup_file name on completeion i.e. the written file is:
# pp_backup.1295886020.tgz
# backup_file=/tmp/pp_backup

# Where do you want the signature docs to be copied, if this is commented 
# out then they will not be copied / extracted.  Note that extracting them 
# will add considerable runtime to pulledpork.
# docs=/path/to/base/www

# The following option, state_order, allows you to more finely control the order
# that pulledpork performs the modify operations, specifically the enablesid
# disablesid and dropsid functions.  An example use case here would be to
# disable an entire category and later enable only a rule or two out of it.
# the valid values are disable, drop, and enable.
# state_order=disable,drop,enable


# Define the path to the pid files of any running process that you want to
# sent a signal (specified with -H option) after PP has completed its run.
#pid_path=/var/run/snort.pid,/var/run/barnyard.pid,/var/run/barnyard2.pid
# and so on...
pid_path=/var/log/snort/snort.pid

# This defines the version of snort that you are using, for use ONLY if the 
# proper snort binary is not on the system that you are fetching the rules with
# This value MUST contain all 4 minor version
# numbers. ET rules are now also dependant on this, verify supported ET versions
# prior to simply throwing rubbish in this variable kthx!
#
# Suricata users - set this to 'suricata-5.x.x' to process rule files
# for suricata, this mimics the -S flag on the command line.
# snort_version=2.9.0.0

# Here you can specify what rule modification files to run automatically.
# simply uncomment and specify the apt path.
# enablesid=/usr/local/etc/snort/enablesid.conf
# dropsid=/usr/local/etc/snort/dropsid.conf
# disablesid=/usr/local/etc/snort/disablesid.conf
# modifysid=/usr/local/etc/snort/modifysid.conf

# What is the base ruleset that you want to use, please uncomment to use
# and see the README.RULESETS for a description of the options.  
# Note that setting this value will disable all ET rulesets if you are 
# Running such rulesets
ips_policy=security

####### Remember, a number of these values are optional.. if you don't 
####### need to process so_rules, simply comment out the so_rule section
####### you can also specify -T at runtime to process only GID 1 rules.

version=0.8.0

Thanks in Advance

[da667]

Running into this issue as well. Also noticing that Noah (the dude who wrote the installation guide for Ubuntu) experienced this problem as well:

https://seclists.org/snort/2021/q1/36

Has there been any progress on this?

[shirkdog]

A look at the latest 3.x tarball, and I do see that there are now precompiled shared object rules being distributed. I am not sure the dumping of the stub rules is necessary in 3.x, since the rules are provided, but the distro types are completely different. I had a note in the code to handle some differences, but it looks like the list will have to be different.

[redbaron4]

I have also encountered this issue. There is a large difference between how Snort2 and Snort3 handle and package SO rules. I will try to document my findings here.

Packaging

Under Snort2, SO rules were in a path like /so_rules/precompiled/Centos-7/x86-64/2.9.17.1/ so Distro needed to be Centos-7 and arch was picked up by PP as x86-64

Under Snort3, SO rules are in a path like /so_rules/precompiled/centos-x64/ so Distro is now centos and arch needs to be set to x64. There is no version string here.

Extraction of stubs

In snort2, the config file had a directive like

dynamicdetection directory /usr/lib64/snort_dynamicrules

which told snort where dynamic rules could be found. So PP extracted the so files directly to this location (via sorule_path). The --dump-dynamicrules=X told snort to look in the directory defined in the config and generate stub rule files which were stored at X. So PP used to run this command with /tmp/tha_rules/so_rules to gather rule stubs at that location.

Under snort3, the said configuration option is no longer there. We do have a new --plugin-path option which I think can be used the same way. So assuming we modify PP to get correct path of so files from rule tarball & save those to sorule_path. However in snort3 --dump-dynamic-rules does not take any options. It works by reading any loaded rules and dumps the stubs to stdout. So to work with this, we will need to run snort with --plugin-path set to sorule_path and expect all rule stubs to be dumped to stdout.

The command would change to

$Snort_path -c $Snort_config --plugin-path=$Sorules --dump-dynamic-rules

and we will need to collect the output under a new file in /tmp/tha_rules/so_rules.

My Perl is not strong enough to generate a PR for this complicated case.

[shirkdog]

I have created a new repo for snort3 called "pulledpork3". There is some initial code that has been written by someone in the community that will be a good starting point for getting snort3 signature updates working for everyone. Any snort3 issues will be tracked here and only closed when resolved with pulledpork3

[redbaron4]

@shirkdog I have created PR #363 which modifies pulledpork script so that it can be used to dump dynamic rules in both Snort2 and Snort3. I have tested it and it appears to work.

Users must set distro to appropriate values. E.g. for Snort2 if distro=RHEL-7 then for Snort3 that should change to distro=centos for correct rules to be copied & their stubs dumped.

[redbaron4]

I think this can now be closed as #363 has merged. There may be a need to update documentation regarding distros for Snort3 being different from Snort2.

[seanjowen]

I downloaded pulledpork.pl today (July 24, 2021) and had to modify the code to properly handle SO_rules for Snort3 (my flavor is ubuntu). I removed the "(" and ")" around $Distro and $arch on line 333 (else). And I also changed pulledpork.conf with distro=ubuntu.

if ($Snortv3 == 0) {
    $sofile_pat_base = $sofile_pat_base . "($Distro)\/($arch)\/($Snort)\/";
}
else {
    $sofile_pat_base = $sofile_pat_base . "$Distro-$arch\/";
}

[redbaron4]

@seanjowen I tried it on my system with and without the paranthesis (( and )) and both times the rules extracted OK. I am on CentOS-7 (although I manually set the distro to ubuntu to test) and my perl is 5.16.3. Maybe its got something to do with perl versions?

[seanjowen]

@redbaron4, my apologies. I retested and the code is fine. The solution was actually simply changing the distro to "ubuntu" rather than "Ubuntu-18-4" in the pulledpork.conf. In my troubleshooting the code, I believe I was distracted by one of my children and had inadvertently changed both the code and the distro config in the same step, leading me to incorrectly conclude it was the code. I had the proper spot in the code that was causing the error, but I had identified the wrong solution. Thanks for building this script and maintaining it!

[redbaron4]

@seanjowen Thanks for clearing it up! And thanks goes to @shirkdog who is the author/maintainer.