Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Missing member in alias if updated #2590

Closed
C-Duv opened this Issue Jul 31, 2018 · 16 comments

Comments

Projects
None yet
4 participants
@C-Duv
Copy link

C-Duv commented Jul 31, 2018

Some of my network seemed incomplete: by looking at Firewall: Diagnostics: pfTables I can see members are missing.

Steps to reproduce:

  1. Go to Firewall: Aliases: View.
  2. Create the following network aliases:
    • TEST_LAN_1 = 192.168.1.0/24
    • TEST_LAN_2 = 192.168.2.0/24
    • TEST_LAN_3 = 192.168.3.0/24
    • TEST_LAN_4 = 192.168.4.0/24
    • TEST_LAN_5 = 192.168.5.0/24
    • TEST_LAN_x = TEST_LAN_1, TEST_LAN_2, TEST_LAN_3
  3. Update TEST_LAN_x by adding the following two members:
    • TEST_LAN_4
    • TEST_LAN_5
  4. Apply alias changes with the Apply changes button.
  5. Alias listing page should show contains:
Name Type Description Values
TEST_LAN_1 Network(s) 192.168.1.0/24
TEST_LAN_2 Network(s) 192.168.2.0/24
TEST_LAN_3 Network(s) 192.168.3.0/24
TEST_LAN_4 Network(s) 192.168.4.0/24
TEST_LAN_5 Network(s) 192.168.5.0/24
TEST_LAN_x Network(s) TEST_LAN_1, TEST_LAN_2, TEST_LAN_3, TEST_LAN_4, TEST_LAN_5
  1. Go to Firewall: Diagnostics: pfTables page.
  2. Select TEST_LAN_x

Expected result:

Table should be:

IP Address
192.168.1.0/24 🗑️
192.168.2.0/24 🗑️
192.168.3.0/24 🗑️
192.168.4.0/24 🗑️
192.168.5.0/24 🗑️

Actual result:

Table is:

IP Address
192.168.1.0/24 🗑️
192.168.2.0/24 🗑️
192.168.3.0/24 🗑️

Extra step:

  1. Go to Firewall: Aliases: View.
  2. Rename alias TEST_LAN_x to TEST_LAN_x_bis.
  3. Apply alias changes with the Apply changes button.
  4. Go to Firewall: Diagnostics: pfTables page.

Table for TEST_LAN_x is still:

IP Address
192.168.1.0/24 🗑️
192.168.2.0/24 🗑️
192.168.3.0/24 🗑️

Table for (the newly created entry) TEST_LAN_x_bis is:

IP Address
192.168.1.0/24 🗑️
192.168.2.0/24 🗑️
192.168.3.0/24 🗑️
192.168.4.0/24 🗑️
192.168.5.0/24 🗑️
@AdSchellevis

This comment has been minimized.

Copy link
Member

AdSchellevis commented Jul 31, 2018

@C-Duv looking fine at my end at step [7]
image

Are you sure you don't have any other aliases defined which might take longer to process (geoip's, url types, etc)? Alias parsing is handled asynchronous, the configd logging might provide you with more insights (System -> Log files -> Configd)

@C-Duv

This comment has been minimized.

Copy link
Author

C-Duv commented Jul 31, 2018

:( I only have Host, Network or port type aliases (for a total of about 40 aliases).

Also, I found out that adding non-alias member to TEST_LAN_x alias seems to make OPNsense "detects" TEST_LAN_4 and TEST_LAN_5 where added to TEST_LAN_x.

  1. Go to Firewall: Aliases: View.
  2. Rename alias TEST_LAN_x_bis to TEST_LAN_x.
  3. Apply alias changes with the Apply changes button.
  4. Go to Firewall: Diagnostics: pfTables page.
  5. Select TEST_LAN_x

Table for TEST_LAN_x is still:

IP Address
192.168.1.0/24 🗑️
192.168.2.0/24 🗑️
192.168.3.0/24 🗑️
  1. Go to Firewall: Aliases: View.
  2. Update TEST_LAN_x by adding the following member:
    • 8.8.8.8
  3. Apply alias changes with the Apply changes button.
  4. Alias listing page should show contains:
Name Type Description Values
TEST_LAN_1 Network(s) 192.168.1.0/24
TEST_LAN_2 Network(s) 192.168.2.0/24
TEST_LAN_3 Network(s) 192.168.3.0/24
TEST_LAN_4 Network(s) 192.168.4.0/24
TEST_LAN_5 Network(s) 192.168.5.0/24
TEST_LAN_x Network(s) TEST_LAN_1, TEST_LAN_2, TEST_LAN_3, TEST_LAN_4, TEST_LAN_5, 8.8.8.8
  1. Go to Firewall: Diagnostics: pfTables page.
  2. Select TEST_LAN_x

Table is now:

IP Address
8.8.8.8 🗑️
192.168.1.0/24 🗑️
192.168.2.0/24 🗑️
192.168.3.0/24 🗑️
192.168.4.0/24 🗑️
192.168.5.0/24 🗑️

The TEST_LAN_4 and TEST_LAN_5 alias values where then rightfully added to TEST_LAN_x.

@fraenki

This comment has been minimized.

Copy link
Member

fraenki commented Dec 10, 2018

I have the same issue on 18.7.8: after updating an existing alias, the change is not propagated to pf. When looking at Firewall: Diagnostics: pfTables the alias still shows the old value; the file in /var/db/aliastables/ALIASNAME.txt also shows the old content.

@AdSchellevis Any advise how to debug this? Maybe there's a way to reload from console and get more verbose output?

@AdSchellevis

This comment has been minimized.

Copy link
Member

AdSchellevis commented Dec 10, 2018

Just to be sure, you did hit apply and there's enough memory reserved for the tables (firewall advanced)?

When properly applied, the alias content is saved to /usr/local/etc/filter_tables.conf, which will be synced to pf using :

/usr/local/opnsense/scripts/filter/update_tables.py

Errors should normally appear in syslog (backend/general)

@fraenki

This comment has been minimized.

Copy link
Member

fraenki commented Dec 10, 2018

Just to be sure, you did hit apply and there's enough memory reserved for the tables (firewall advanced)?

"Firewall Maximum Table Entries" is set to 5.000.000 now, no change.

When properly applied, the alias content is saved to /usr/local/etc/filter_tables.conf

The change is visible in this file. However, I should note that this is about a nested alias: one alias contains other aliases of type "network".

which will be synced to pf using :
/usr/local/opnsense/scripts/filter/update_tables.py

I was able to run this script without any error. But Firewall: Diagnostics: pfTables still shows the old alias state.

Errors should normally appear in syslog (backend/general)

No errors in system.log nor configd.log. :(

@AdSchellevis

This comment has been minimized.

Copy link
Member

AdSchellevis commented Dec 11, 2018

@fraenki would it be possible to get your alias section? if I can reproduce it here, I can probably fix it.

@fraenki

This comment has been minimized.

Copy link
Member

fraenki commented Dec 11, 2018

@AdSchellevis Sure, here we go. I've constructed a test case of the nested structure and I was able to verify on my test machine that the pf table test_networks will remain empty in this case.

          <alias uuid="5efe9560-da33-4d00-86ba-3660bbd95f85">
            <enabled>1</enabled>
            <name>net_test1</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.1.0/24</content>
            <description/>
          </alias>
          <alias uuid="7c259a94-bfbb-4df3-9aeb-fd3d61e07fb9">
            <enabled>1</enabled>
            <name>net_test2</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.2.0/24</content>
            <description/>
          </alias>
          <alias uuid="0c84b625-393f-4bc4-88e7-7b4cd281d5da">
            <enabled>1</enabled>
            <name>net_test3</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.3.0/24</content>
            <description/>
          </alias>
          <alias uuid="a1b4936d-eb25-4430-96bc-f7d09b5dd8b0">
            <enabled>1</enabled>
            <name>net_test4</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.4.0/24</content>
            <description/>
          </alias>
          <alias uuid="2add0047-3970-4374-aabe-3ecf0f69df96">
            <enabled>1</enabled>
            <name>net_test5</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.5.0/24</content>
            <description/>
          </alias>
          <alias uuid="b8cb19d2-5e99-4f10-8290-2f4185553f37">
            <enabled>1</enabled>
            <name>net_test6</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.6.0/24</content>
            <description/>
          </alias>
          <alias uuid="e2843b51-49f5-4a00-bdb9-c3643ae2bdd9">
            <enabled>1</enabled>
            <name>net_test7</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.7.0/24</content>
            <description/>
          </alias>
          <alias uuid="7fb21727-6928-42d8-9bf0-3615ae7dda1e">
            <enabled>1</enabled>
            <name>net_test8</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.8.0/24</content>
            <description/>
          </alias>
          <alias uuid="714d3b61-1232-4a91-9f88-b016c686b32e">
            <enabled>1</enabled>
            <name>net_test9</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.0.9.0/24</content>
            <description/>
          </alias>

          <alias uuid="33b9c219-bcbd-4629-a54a-721ad76f2ee9">
            <enabled>1</enabled>
            <name>net_foo_bar1</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.1.0/24</content>
            <description/>
          </alias>
          <alias uuid="4f4c4731-3b35-4392-81ea-70b69f62c186">
            <enabled>1</enabled>
            <name>net_foo_bar2</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.2.0/24</content>
            <description/>
          </alias>
          <alias uuid="572f4031-6917-461b-9376-6bb2f3b8767f">
            <enabled>1</enabled>
            <name>net_foo_bar3</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.3.0/24</content>
            <description/>
          </alias>
          <alias uuid="9475ba97-d27f-4ba6-842c-6f9488b763f2">
            <enabled>1</enabled>
            <name>net_foo_bar4</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>10.99.4.0/24</content>
            <description/>
          </alias>

          <alias uuid="94bb5c81-6a21-4679-a00b-60201498c9c4">
            <enabled>1</enabled>
            <name>test_networks</name>
            <type>network</type>
            <proto/>
            <updatefreq/>
            <content>net_test1
net_test2
net_test3
net_test4
net_test5
net_test6
net_test7
net_test8
net_test9
net_foo_bar1
net_foo_bar2
net_foo_bar3
net_foo_bar4</content>
            <description>test foo </description>
          </alias>

(spaces added to better distinct sections)

@fraenki

This comment has been minimized.

Copy link
Member

fraenki commented Dec 11, 2018

@AdSchellevis One more addition: Apparently this seems to be related to aliases of type GeoIP. If I disable the only GeoIP alias, then the pf tables will be properly populated. It seems that GeoIP aliases break things. How to go on from here?

@AdSchellevis

This comment has been minimized.

Copy link
Member

AdSchellevis commented Dec 11, 2018

@fraenki if you ls the alias directory, does it change? Maybe it's active and locked
ls -als /var/db/aliastables/

@fraenki

This comment has been minimized.

Copy link
Member

fraenki commented Dec 11, 2018

@AdSchellevis Indeed, an ls -alsrt /var/db/aliastables/ looks strange: the test_networks file from my test is not updated at all; I've tried multiple changes and hitting "Apply" several times. Besides that the GeoIP alias file is empty. I've tried to create a new GeoIP alias, but only the ALIASNAME.self.txt file is created, nothing else, the other files (for the GeoIP alias) are missing completely.

Not sure why, but I can finally see an error regarding aliases in configd.log:

Dec 11 23:05:03 opnsense-test configd.py: [c8e51cef-33d0-47ad-ac45-28e7d4f71585] generate template OPNsense/Filter
Dec 11 23:05:04 opnsense-test configd.py: generate template container OPNsense/Filter
Dec 11 23:05:04 opnsense-test configd.py:  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
Dec 11 23:05:04 opnsense-test configd.py: [446d42f7-5a91-44f2-9c8a-6409ece79273] Reloading filter
Dec 11 23:05:06 opnsense-test configd.py: [c9da607d-93c9-4594-aa5b-e4bfa802aa66] generate template OPNsense/Filter
Dec 11 23:05:06 opnsense-test configd.py: generate template container OPNsense/Filter
Dec 11 23:05:06 opnsense-test configd.py:  OPNsense/Filter generated //usr/local/etc/filter_tables.conf
Dec 11 23:05:06 opnsense-test configd.py: [0c66392c-c68f-435a-a1f3-3123bc0f5e77] refresh url table aliases
Dec 11 23:05:06 opnsense-test configd.py: [0777cb0c-e642-4209-b61e-76db9a998d5d] refresh url table aliases
Dec 11 23:05:08 opnsense-test configd.py: [0c66392c-c68f-435a-a1f3-3123bc0f5e77] returned exit status 1
Dec 11 23:05:08 opnsense-test configd.py: message 0c66392c-c68f-435a-a1f3-3123bc0f5e77 [filter.refresh_aliases] returned Error (1)  

I'm not 100% sure, but I think this error appeared for the first time when I decided to remove my first GeoIP alias and created a new one.

@AdSchellevis

This comment has been minimized.

Copy link
Member

AdSchellevis commented Dec 12, 2018

hmm, it suggest it crashes out on "refresh_aliases",

ok, next steps:

  1. does /tmp/filter_update_tables.lock exist at this time?
  2. is it running (ps fax | grep update_tables)?
  3. if both no at this time, what does /usr/local/opnsense/scripts/filter/update_tables.py output when run from the console?
@fraenki

This comment has been minimized.

Copy link
Member

fraenki commented Dec 12, 2018

  1. file exists, but with a really old timestamp:
root@opnsense-test:~ # ls -l /tmp/filter_update_tables.lock
-rw-r-----  1 root  wheel  0 Nov 22 16:27 /tmp/filter_update_tables.lock
  1. nope, no process found

  2. ah, that's interesting:

root@opnsense-test:~ # /usr/local/opnsense/scripts/filter/update_tables.py
Traceback (most recent call last):
  File "/usr/local/opnsense/scripts/filter/update_tables.py", line 122, in <module>
    alias_content = alias.resolve()
  File "/usr/local/opnsense/scripts/filter/lib/alias.py", line 236, in resolve
    for address in address_parser(item):
  File "/usr/local/opnsense/scripts/filter/lib/alias.py", line 171, in _fetch_geo
    for proto in self._proto.split(','):
AttributeError: 'NoneType' object has no attribute 'split'
@AdSchellevis

This comment has been minimized.

Copy link
Member

AdSchellevis commented Dec 12, 2018

duplicate #2986 fix a8b0c06

@fichtner

This comment has been minimized.

Copy link
Member

fichtner commented Dec 12, 2018

it's in 18.7.9 then...

@fraenki

This comment has been minimized.

Copy link
Member

fraenki commented Dec 13, 2018

Applied a8b0c06 manually and it fixes the bug. Thanks, guys! (I'd close this issue, but I'm not the original reporter.)

@fichtner

This comment has been minimized.

Copy link
Member

fichtner commented Dec 13, 2018

ok, great. thanks for confirming!

@fichtner fichtner closed this Dec 13, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.