Missing member in alias if updated #2590

C-Duv · 2018-07-31T09:28:13Z

Some of my network seemed incomplete: by looking at Firewall: Diagnostics: pfTables I can see members are missing.

Steps to reproduce:

Go to Firewall: Aliases: View.
Create the following network aliases:
- TEST_LAN_1 = 192.168.1.0/24
- TEST_LAN_2 = 192.168.2.0/24
- TEST_LAN_3 = 192.168.3.0/24
- TEST_LAN_4 = 192.168.4.0/24
- TEST_LAN_5 = 192.168.5.0/24
- TEST_LAN_x = TEST_LAN_1, TEST_LAN_2, TEST_LAN_3
Update TEST_LAN_x by adding the following two members:
- TEST_LAN_4
- TEST_LAN_5
Apply alias changes with the Apply changes button.
Alias listing page should show contains:

Name	Type	Values
TEST_LAN_1	Network(s)	192.168.1.0/24
TEST_LAN_2	Network(s)	192.168.2.0/24
TEST_LAN_3	Network(s)	192.168.3.0/24
TEST_LAN_4	Network(s)	192.168.4.0/24
TEST_LAN_5	Network(s)	192.168.5.0/24
TEST_LAN_x	Network(s)	TEST_LAN_1, TEST_LAN_2, TEST_LAN_3, TEST_LAN_4, TEST_LAN_5

Go to Firewall: Diagnostics: pfTables page.
Select TEST_LAN_x

Expected result:

Table should be:

IP Address
192.168.1.0/24	🗑️
192.168.2.0/24	🗑️
192.168.3.0/24	🗑️
192.168.4.0/24	🗑️ ️
192.168.5.0/24	🗑️

Actual result:

Table is:

IP Address
192.168.1.0/24	🗑️
192.168.2.0/24	🗑️
192.168.3.0/24	🗑️

Extra step:

Go to Firewall: Aliases: View.
Rename alias TEST_LAN_x to TEST_LAN_x_bis.
Apply alias changes with the Apply changes button.
Go to Firewall: Diagnostics: pfTables page.

Table for TEST_LAN_x is still:

IP Address
192.168.1.0/24	🗑️
192.168.2.0/24	🗑️
192.168.3.0/24	🗑️

Table for (the newly created entry) TEST_LAN_x_bis is:

IP Address
192.168.1.0/24	🗑️
192.168.2.0/24	🗑️
192.168.3.0/24	🗑️
192.168.4.0/24	🗑️ ️
192.168.5.0/24	🗑️

The text was updated successfully, but these errors were encountered:

AdSchellevis · 2018-07-31T09:40:22Z

@C-Duv looking fine at my end at step [7]

Are you sure you don't have any other aliases defined which might take longer to process (geoip's, url types, etc)? Alias parsing is handled asynchronous, the configd logging might provide you with more insights (System -> Log files -> Configd)

C-Duv · 2018-07-31T09:43:51Z

:( I only have Host, Network or port type aliases (for a total of about 40 aliases).

Also, I found out that adding non-alias member to TEST_LAN_x alias seems to make OPNsense "detects" TEST_LAN_4 and TEST_LAN_5 where added to TEST_LAN_x.

Go to Firewall: Aliases: View.
Rename alias TEST_LAN_x_bis to TEST_LAN_x.
Apply alias changes with the Apply changes button.
Go to Firewall: Diagnostics: pfTables page.
Select TEST_LAN_x

Table for TEST_LAN_x is still:

IP Address
192.168.1.0/24	🗑️
192.168.2.0/24	🗑️
192.168.3.0/24	🗑️

Go to Firewall: Aliases: View.
Update TEST_LAN_x by adding the following member:
- 8.8.8.8
Apply alias changes with the Apply changes button.
Alias listing page should show contains:

Name	Type	Values
TEST_LAN_1	Network(s)	192.168.1.0/24
TEST_LAN_2	Network(s)	192.168.2.0/24
TEST_LAN_3	Network(s)	192.168.3.0/24
TEST_LAN_4	Network(s)	192.168.4.0/24
TEST_LAN_5	Network(s)	192.168.5.0/24
TEST_LAN_x	Network(s)	TEST_LAN_1, TEST_LAN_2, TEST_LAN_3, TEST_LAN_4, TEST_LAN_5, 8.8.8.8

Go to Firewall: Diagnostics: pfTables page.
Select TEST_LAN_x

Table is now:

IP Address
8.8.8.8	🗑️
192.168.1.0/24	🗑️
192.168.2.0/24	🗑️
192.168.3.0/24	🗑️
192.168.4.0/24	🗑️ ️
192.168.5.0/24	🗑️

The TEST_LAN_4 and TEST_LAN_5 alias values where then rightfully added to TEST_LAN_x.

fraenki · 2018-12-10T21:33:42Z

I have the same issue on 18.7.8: after updating an existing alias, the change is not propagated to pf. When looking at Firewall: Diagnostics: pfTables the alias still shows the old value; the file in /var/db/aliastables/ALIASNAME.txt also shows the old content.

@AdSchellevis Any advise how to debug this? Maybe there's a way to reload from console and get more verbose output?

AdSchellevis · 2018-12-10T21:47:15Z

Just to be sure, you did hit apply and there's enough memory reserved for the tables (firewall advanced)?

When properly applied, the alias content is saved to /usr/local/etc/filter_tables.conf, which will be synced to pf using :

/usr/local/opnsense/scripts/filter/update_tables.py

Errors should normally appear in syslog (backend/general)

fraenki · 2018-12-10T22:22:34Z

Just to be sure, you did hit apply and there's enough memory reserved for the tables (firewall advanced)?

"Firewall Maximum Table Entries" is set to 5.000.000 now, no change.

When properly applied, the alias content is saved to /usr/local/etc/filter_tables.conf

The change is visible in this file. However, I should note that this is about a nested alias: one alias contains other aliases of type "network".

which will be synced to pf using :
/usr/local/opnsense/scripts/filter/update_tables.py

I was able to run this script without any error. But Firewall: Diagnostics: pfTables still shows the old alias state.

Errors should normally appear in syslog (backend/general)

No errors in system.log nor configd.log. :(

AdSchellevis · 2018-12-11T07:47:57Z

@fraenki would it be possible to get your alias section? if I can reproduce it here, I can probably fix it.

fraenki · 2018-12-11T16:23:21Z

@AdSchellevis Sure, here we go. I've constructed a test case of the nested structure and I was able to verify on my test machine that the pf table test_networks will remain empty in this case.

          <alias uuid="5efe9560-da33-4d00-86ba-3660bbd95f85">
            <enabled>1</enabled>
            <name>net_test1</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.1.0/24</content>
            <description/>
          </alias>
          <alias uuid="7c259a94-bfbb-4df3-9aeb-fd3d61e07fb9">
            <enabled>1</enabled>
            <name>net_test2</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.2.0/24</content>
            <description/>
          </alias>
          <alias uuid="0c84b625-393f-4bc4-88e7-7b4cd281d5da">
            <enabled>1</enabled>
            <name>net_test3</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.3.0/24</content>
            <description/>
          </alias>
          <alias uuid="a1b4936d-eb25-4430-96bc-f7d09b5dd8b0">
            <enabled>1</enabled>
            <name>net_test4</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.4.0/24</content>
            <description/>
          </alias>
          <alias uuid="2add0047-3970-4374-aabe-3ecf0f69df96">
            <enabled>1</enabled>
            <name>net_test5</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.5.0/24</content>
            <description/>
          </alias>
          <alias uuid="b8cb19d2-5e99-4f10-8290-2f4185553f37">
            <enabled>1</enabled>
            <name>net_test6</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.6.0/24</content>
            <description/>
          </alias>
          <alias uuid="e2843b51-49f5-4a00-bdb9-c3643ae2bdd9">
            <enabled>1</enabled>
            <name>net_test7</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.7.0/24</content>
            <description/>
          </alias>
          <alias uuid="7fb21727-6928-42d8-9bf0-3615ae7dda1e">
            <enabled>1</enabled>
            <name>net_test8</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.8.0/24</content>
            <description/>
          </alias>
          <alias uuid="714d3b61-1232-4a91-9f88-b016c686b32e">
            <enabled>1</enabled>
            <name>net_test9</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.9.0/24</content>
            <description/>
          </alias>

          <alias uuid="33b9c219-bcbd-4629-a54a-721ad76f2ee9">
            <enabled>1</enabled>
            <name>net_foo_bar1</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.1.0/24</content>
            <description/>
          </alias>
          <alias uuid="4f4c4731-3b35-4392-81ea-70b69f62c186">
            <enabled>1</enabled>
            <name>net_foo_bar2</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.2.0/24</content>
            <description/>
          </alias>
          <alias uuid="572f4031-6917-461b-9376-6bb2f3b8767f">
            <enabled>1</enabled>
            <name>net_foo_bar3</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.3.0/24</content>
            <description/>
          </alias>
          <alias uuid="9475ba97-d27f-4ba6-842c-6f9488b763f2">
            <enabled>1</enabled>
            <name>net_foo_bar4</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.4.0/24</content>
            <description/>
          </alias>

          <alias uuid="94bb5c81-6a21-4679-a00b-60201498c9c4">
            <enabled>1</enabled>
            <name>test_networks</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>net_test1
net_test2
net_test3
net_test4
net_test5
net_test6
net_test7
net_test8
net_test9
net_foo_bar1
net_foo_bar2
net_foo_bar3
net_foo_bar4</content>
            <description>test foo </description>
          </alias>

(spaces added to better distinct sections)

fraenki · 2018-12-11T16:26:54Z

@AdSchellevis One more addition: Apparently this seems to be related to aliases of type GeoIP. If I disable the only GeoIP alias, then the pf tables will be properly populated. It seems that GeoIP aliases break things. How to go on from here?

AdSchellevis · 2018-12-11T20:40:22Z

@fraenki if you ls the alias directory, does it change? Maybe it's active and locked
ls -als /var/db/aliastables/

fraenki · 2018-12-11T22:08:24Z

@AdSchellevis Indeed, an ls -alsrt /var/db/aliastables/ looks strange: the test_networks file from my test is not updated at all; I've tried multiple changes and hitting "Apply" several times. Besides that the GeoIP alias file is empty. I've tried to create a new GeoIP alias, but only the ALIASNAME.self.txt file is created, nothing else, the other files (for the GeoIP alias) are missing completely.

Not sure why, but I can finally see an error regarding aliases in configd.log:

Dec 11 23:05:03 opnsense-test configd.py: [c8e51cef-33d0-47ad-ac45-28e7d4f71585] generate template OPNsense/Filter
Dec 11 23:05:04 opnsense-test configd.py: generate template container OPNsense/Filter
Dec 11 23:05:04 opnsense-test configd.py:  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
Dec 11 23:05:04 opnsense-test configd.py: [446d42f7-5a91-44f2-9c8a-6409ece79273] Reloading filter
Dec 11 23:05:06 opnsense-test configd.py: [c9da607d-93c9-4594-aa5b-e4bfa802aa66] generate template OPNsense/Filter
Dec 11 23:05:06 opnsense-test configd.py: generate template container OPNsense/Filter
Dec 11 23:05:06 opnsense-test configd.py:  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
Dec 11 23:05:06 opnsense-test configd.py: [0c66392c-c68f-435a-a1f3-3123bc0f5e77] refresh url table aliases
Dec 11 23:05:06 opnsense-test configd.py: [0777cb0c-e642-4209-b61e-76db9a998d5d] refresh url table aliases
Dec 11 23:05:08 opnsense-test configd.py: [0c66392c-c68f-435a-a1f3-3123bc0f5e77] returned exit status 1
Dec 11 23:05:08 opnsense-test configd.py: message 0c66392c-c68f-435a-a1f3-3123bc0f5e77 [filter.refresh_aliases] returned Error (1)

I'm not 100% sure, but I think this error appeared for the first time when I decided to remove my first GeoIP alias and created a new one.

AdSchellevis · 2018-12-12T07:46:33Z

hmm, it suggest it crashes out on "refresh_aliases",

ok, next steps:

does /tmp/filter_update_tables.lock exist at this time?
is it running (ps fax | grep update_tables)?
if both no at this time, what does /usr/local/opnsense/scripts/filter/update_tables.py output when run from the console?

fraenki · 2018-12-12T12:32:45Z

file exists, but with a really old timestamp:

root@opnsense-test:~ # ls -l /tmp/filter_update_tables.lock
-rw-r-----  1 root  wheel  0 Nov 22 16:27 /tmp/filter_update_tables.lock

nope, no process found
ah, that's interesting:

root@opnsense-test:~ # /usr/local/opnsense/scripts/filter/update_tables.py
Traceback (most recent call last):
  File "/usr/local/opnsense/scripts/filter/update_tables.py", line 122, in <module>
    alias_content = alias.resolve()
  File "/usr/local/opnsense/scripts/filter/lib/alias.py", line 236, in resolve
    for address in address_parser(item):
  File "/usr/local/opnsense/scripts/filter/lib/alias.py", line 171, in _fetch_geo
    for proto in self._proto.split(','):
AttributeError: 'NoneType' object has no attribute 'split'

AdSchellevis · 2018-12-12T17:08:02Z

duplicate #2986 fix a8b0c06

fichtner · 2018-12-12T17:20:07Z

it's in 18.7.9 then...

fraenki · 2018-12-13T09:34:26Z

Applied a8b0c06 manually and it fixes the bug. Thanks, guys! (I'd close this issue, but I'm not the original reporter.)

fichtner · 2018-12-13T09:36:23Z

ok, great. thanks for confirming!

AdSchellevis added the support Community support label Jul 31, 2018

fichtner closed this as completed Dec 13, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Missing member in alias if updated #2590

Missing member in alias if updated #2590

C-Duv commented Jul 31, 2018 •

edited

Loading

AdSchellevis commented Jul 31, 2018

C-Duv commented Jul 31, 2018 •

edited

Loading

fraenki commented Dec 10, 2018

AdSchellevis commented Dec 10, 2018

fraenki commented Dec 10, 2018

AdSchellevis commented Dec 11, 2018

fraenki commented Dec 11, 2018

fraenki commented Dec 11, 2018

AdSchellevis commented Dec 11, 2018

fraenki commented Dec 11, 2018 •

edited

Loading

AdSchellevis commented Dec 12, 2018

fraenki commented Dec 12, 2018

AdSchellevis commented Dec 12, 2018

fichtner commented Dec 12, 2018

fraenki commented Dec 13, 2018

fichtner commented Dec 13, 2018

Missing member in alias if updated #2590

Missing member in alias if updated #2590

Comments

C-Duv commented Jul 31, 2018 • edited Loading

AdSchellevis commented Jul 31, 2018

C-Duv commented Jul 31, 2018 • edited Loading

fraenki commented Dec 10, 2018

AdSchellevis commented Dec 10, 2018

fraenki commented Dec 10, 2018

AdSchellevis commented Dec 11, 2018

fraenki commented Dec 11, 2018

fraenki commented Dec 11, 2018

AdSchellevis commented Dec 11, 2018

fraenki commented Dec 11, 2018 • edited Loading

AdSchellevis commented Dec 12, 2018

fraenki commented Dec 12, 2018

AdSchellevis commented Dec 12, 2018

fichtner commented Dec 12, 2018

fraenki commented Dec 13, 2018

fichtner commented Dec 13, 2018

C-Duv commented Jul 31, 2018 •

edited

Loading

C-Duv commented Jul 31, 2018 •

edited

Loading

fraenki commented Dec 11, 2018 •

edited

Loading