Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] / [Feedback] Extension of the NMAP functionality. #271

Closed
accessiblepixel opened this issue Mar 11, 2024 · 15 comments
Closed

Comments

@accessiblepixel
Copy link

accessiblepixel commented Mar 11, 2024

Hi!

Many thanks for your excellent fork, I'm very impressed with it so far.

I've encountered a few issues so far, but nothing so serious that I couldn't work around it.

My network is separated into segmented VLANs so the default arp scanning can only access the segment pialert is assigned to. To work around this, and because pihole and pialert are hosted in a Proxmox LXC, I added interfaces for each VLAN without an IP address, for example eth0 is my default interface that all the other network communication happens through, and eth5 is for VLAN 5 is a segmented vlan.

Because I didn't want to actually have the lxc accessible on each of these VLANs via an IP address I didn't add one to any other VLAN interfaces, but I found in the documentation for arpscan I can specify the reply address as an IP somewhere on the VLAN - As far as I am aware this will allow devices that check if an arp request is on the same subnet to reply - without specifying this it uses the (invalid) source address 0.0.0.0 (which some devices take issue with)

So in the pialert.conf file I have
SCAN_SUBNETS = ['192.168.0.1/24 --interface=eth0','192.168.5.0/24 --interface=eth5 --arpspa=192.168.5.253']

This has now shown up most devices on my network :)

Now to my question - The settings for Speedtest. The option for SPEEDTEST_TASK_HOUR - what does this specify, the hour in which a speedtest will be run?

Also, does the speedtest-cli need to be installed from ookla to the system or is the one included already with the pi.alert distribution ( pialert/back/speedtest-cli) enough?

Many thanks for the amazing work you've put into maintaining this fork so far.

If I've put this in the wrong place, please feel free to move it to where is more appropriate - I'm not particularly used to github practices yet (and they can vary per repo)

Hope you're having a great day.
Kind regards!

Edit: I was also wondering if there was some way to schedule automatic nmap scans and have the results stored in the DB?

@leiweibau
Copy link
Owner

leiweibau commented Mar 12, 2024

Because I didn't want to actually have the lxc accessible on each of these VLANs via an IP address I didn't add one to any other VLAN interfaces, but I found in the documentation for arpscan I can specify the reply address as an IP somewhere on the VLAN - As far as I am aware this will allow devices that check if an arp request is on the same subnet to reply - without specifying this it uses the (invalid) source address 0.0.0.0 (which some devices take issue with)

Nice trick. As this is probably a slightly more unusual configuration, I'm still undecided whether to include this in the documentation so as not to cause any additional confusion.

The settings for Speedtest. The option for SPEEDTEST_TASK_HOUR - what does this specify, the hour in which a speedtest will be run?

Yes, the full hour at which the speed test is performed is specified here. 9 = 9am, 14 = 2pm (14 o'clock). However, the Ookla Speedtest must be "installed" for this (https://raw.githubusercontent.com/leiweibau/Pi.Alert/assets/screen_dev_internet_01.png)

Also, does the speedtest-cli need to be installed from ookla to the system or is the one included already with the pi.alert distribution ( pialert/back/speedtest-cli) enough?

The speedtest-cli will be removed sooner or later. The background is that there was once an issue with this client (Starlink user). However, the official binary from Ookla worked correctly. The last update should also fix the bug that you have to re-download the Ookla client after every Pi.Alert update.

Thank you for the praise and kind words

Addendum:

I was also wondering if there was some way to schedule automatic nmap scans and have the results stored in the DB?

I had thought about it once, but rejected it for the time being, because depending on the configuration, a nmap scan can take quite a while and I don't want to have a race condition on the database.
My last thoughts in this direction were to write the results of a manual scan to the database and, if possible, to compare them with an already existing scan. With this commit (9945dc7), among others, I have already laid the foundation for further processing of the nmap output. I have not yet decided when and, above all, how I will implement it. What is the point of having all kinds of data if it is presented in a confusing way? I'm not a UI designer, but I try to make my fork as organized as possible.

@Tuxtom007
Copy link

Thanks for the tip with "arpspa", but unfortunately it did not solve my problem that the VLANs are scanned.

However, an "arp-scan 10.0.10.0/24 --interface=eth1" to the corresponding VLAN still works without any problems, which also worked before with my various configuration attempts, just not from the PiAlert

@accessiblepixel
Copy link
Author

@leiweibau 'unusual network configuration' - yes please! :D

I imagine my configuration won't be anywhere near the standard. I just ended up segmenting everything out as VLANs, mainly for organisation in my head- Oh, IP is 192.168.5.x I know what "group" it's in, VLAN5 and it's level of trust. I'm not sure if LXC can have one network interface that has multiple tagged VLANs to it (read docs for a while... it says maybe), so I just kinda "brute forced" it by adding an interface for each VLAN I used!

I didn't wanna have a bunch of random IPs on each one though, since if you've got 10 VLANs, that's 10 extra IPs - which would probably appear in arpscan too lol

Thanks for the hints about speedtest - I installed the speedtest apt repo and the client, and now things seem to be working. I also figured you can specify multiple hours - at first I thought it was just one. Although might it be a good idea to randomise the minute/second that the automated scan is done? - I vaguely remember as a young sysadmin being taught that automation on the hour for other people's systems is a bad thing 🤣

RE: Nmap - My thinking with it was maybe a scan could be done over a predefined period, not that often and even if it's just as the text output is now it can display the "cached" nmap results when you click on to it... although guess that won't do anything for "monitoring changes"... 🤷‍♀️

Though I do like the idea of doing the scan, getting the results and once they've been "got" adding to the database - sounds like a sensible way to do it to me! :) Won't have to worry about locks too much then.

@Tuxtom007 I'm not sure if arpspa is required or not, or if it just needs a network interface on each VLAN. I guess I could remove arpspa and see if I lose any devices but I'm hesitant to fix what ain't broke for me. 🤪

Hope you're both doing well. :)

@leiweibau
Copy link
Owner

I vaguely remember as a young sysadmin being taught that automation on the hour for other people's systems is a bad thing 🤣

Since the users of my fork are a bit more internationally dispersed and Ookla certainly doesn't just have the one speedtest server, that should still be okay 😉 Randomizing the minutes in the relevant hour is not as trivial to implement as it might seem at first. I'll take it up as an idea, but without the guarantee that I will implement it.

My thinking with it was maybe a scan could be done over a predefined period, not that often

There are actually 2 things to address here:

  1. a cyclic nmap scan of the network as a addition to the ARP scan:
    Depending on the size of the network, there will inevitably be a race condition here, as the scan is "cron based", i.e. it starts every 5 min.
  2. a cyclic scan on one or more devices:
    Even with the objective of doing this only for selected devices, there will be the one user who activates the function for all, or at least for many devices and then gets frustrated that nothing works anymore and Pi.Alert generates lots of error messages. I don't really need that 😉

For these reasons I have decided against automatic nmap scans

@accessiblepixel
Copy link
Author

@leiweibau

I understand the reasoning, although one thing to say about things that run with cron of undetermined length, a flag set "is_running" could be set so if it's called again while one is in progress... I have do that on microprocessors all the time :) Again there's the issue if that gets set and then things don't work and I can see why you don't want the added support overhead...

Would you consider adding caching of last nmap result? Since that's probably enough for my own personal needs.

Anyway, thanks for your replies and work so far. My suggestions are mearly that, any you think are valuable to use going forward, please do so, any that make no sense ... you don't gotta use haha :)

@leiweibau
Copy link
Owner

a flag set "is_running" could be set so if it's called again while one is in progress

good point 🤔 and so obviously

@leiweibau
Copy link
Owner

Would you consider adding caching of last nmap result?

I think that something will happen here, as I have already done the groundwork. I just have to think about how to implement it properly.

@leiweibau leiweibau changed the title Some ideas/Feedback - My experience with vlans and a question :) [Feature Request] / [Feedback] Extension of the NMAP functionality. Mar 13, 2024
@Tuxtom007
Copy link

Tuxtom007 commented Mar 13, 2024

@accessiblepixel

I have the same constellation, Pialtert runs on a LXC in Proxmox, I use 12 VLANs in my network and have given the PiAlert-LXC an additional network card in the first 3 VLANs as a test (each also with a static IP in the VLAN via DHCP reservation).
An arpscan on the LXC directly with the network and the Inetrface works perfectly, I get all clients displayed as the 3 VLAN respectively.

But no matter how I enter it in pialert.conf, I don't get any clients from the other VLANs. I have already tried all possible combinations from the documentation: IP network + interface + VLAN, then without VLAN, only IP network + VLAN, etc.

I used to run PiAlert in the same way some time ago, it was a real pain to get all VLANs scanned, but unfortunately I can't remember how I did it.

Yesterday I switched to another fork, let's see which is better, the WebUI at least looks different and there is also the option to get the client from the Unifi controller.

leiweibau added a commit that referenced this issue Mar 19, 2024
- Remove nmap session log button from settings page
- Remove nmap modal box
- Update language files (scan infos moved to faq/help page)
- insert nmap scan results to db
- changed scan results presentation
- set timeout for nmap scan

Regarding to #271
@leiweibau
Copy link
Owner

leiweibau commented Mar 20, 2024

This is the current status of my work.
If there is already a saved "nmap scan", this is displayed below the buttons

one

If a new scan has been performed, the two results are displayed next to each other. As soon as the page has been reloaded, only the most recent scan is displayed again. In the case that the database is currently being locked, there is currently no fallback solution as with the journal.

two

A cleanup is also run over this table, i.e. old entries are deleted regularly. I plan to add a button in the "Tools" tab to save all available nmap scans for a device

leiweibau added a commit that referenced this issue Mar 21, 2024
- Maintenance effort for outputting scan results reduced
- Help files of the various language files updated

Regarding to #271
leiweibau added a commit that referenced this issue Mar 22, 2024
- Remove/streamline old code
- Prepare csv download for nmap scan per device
- Replace "static strings" with multi-language strings
- update language files

Regarding to #271
leiweibau added a commit that referenced this issue Mar 22, 2024
- finish CSV download
- journal will follow

Regarding to #271
leiweibau added a commit that referenced this issue Mar 26, 2024
New:
* Add "Delete all nmap scans" Button to maintenance page
* Button to reset the search on the journal page
* CSV download for nmap scan per device
* Insert Nmap scan results to db (only scans with results are saved in the db) (#271)

Fixed:
* Minor fixes and tweaks
* pialert-cli help: When the end of the command list is reached, the command prompt appears again automatically

Updated:
* pialert-cli: to update the database for the new function
* Readme: add Sponsor
* Readme: LINUX-DISTRIBUTIONS.md (Issue Report via mail)
* Cleanup task extended (Nmap scan results)
* Remove/streamline old code
* Language files

Changed:
* Set timeout (60s )for all Nmap scan types
* Update Check: Check for a special installation directory to avoid issues like #283 or #281
* Nmap scan infos moved to faq/help page

Removed:
* Remove Nmap session log button from settings page
leiweibau added a commit that referenced this issue Mar 28, 2024
New:
* Add "Delete all nmap scans" Button to maintenance tab
* Add "Delete all Webservices" Button to maintenance tab
* Button to reset the search on the journal page
* CSV download for nmap scan per device
* Insert Nmap scan results to db (only scans with results are saved in the db) (#271)

Fixed:
* Minor fixes and tweaks
* pialert-cli help: When the end of the command list is reached, the command prompt appears again automatically
* Fixed redirections in the event of an error (reported via mail)

Updated:
* pialert-cli: to update the database for the new function
* Readme: add Sponsor
* Readme: LINUX-DISTRIBUTIONS.md (reported via mail)
* Cleanup task extended (Nmap scan results)
* Remove/streamline old code
* Language files

Changed:
* Set timeout (60s )for all Nmap scan types
* The counter of the currently active scans in the status box (settings page) has been replaced by a simple display (Scans running / No Scans running). Depending on the Linux distribution, the counter varied by +/- 1. The original reason for the introduction of the counter was a ramping up of arp-scan processes, which has not been a problem for many Pi.Alert versions.
* Update Check: Check for a special installation directory to avoid issues like #283 or #281
* Nmap scan infos moved to faq/help page

Removed:
* Remove Nmap session log button from settings page
leiweibau added a commit that referenced this issue Mar 28, 2024
New:
* Add "Delete all nmap scans" Button to maintenance tab
* Add "Delete all Webservices" Button to maintenance tab
* Button to reset the search on the journal page
* CSV download for nmap scan per device
* Insert Nmap scan results to db (only scans with results are saved in the db) (#271)

Fixed:
* Minor fixes and tweaks
* pialert-cli help: When the end of the command list is reached, the command prompt appears again automatically
* Fixed redirections in the event of an error (reported via mail)

Updated:
* pialert-cli: to update the database for the new function
* Readme: add Sponsor
* Readme: LINUX-DISTRIBUTIONS.md (reported via mail)
* Cleanup task extended (Nmap scan results)
* Remove/streamline old code
* Language files

Changed:
* Set timeout (60s )for all Nmap scan types
* The counter of the currently active scans in the status box (settings page) has been replaced by a simple display (Scans running / No Scans running). Depending on the Linux distribution, the counter varied by +/- 1. The original reason for the introduction of the counter was a ramping up of arp-scan processes, which has not been a problem for many Pi.Alert versions.
* Update Check: Check for a special installation directory to avoid issues like #283 or #281
* Nmap scan infos moved to faq/help page

Removed:
* Remove Nmap session log button from settings page
@leiweibau
Copy link
Owner

Update released with b3da81c

@leiweibau
Copy link
Owner

The manual Nmap scan function has been significantly expanded. Other aspects of the request will not be implemented for the time being, as such changes would have far-reaching consequences that need to be weighed up.
For these reasons, this request is now closed.

@accessiblepixel
Copy link
Author

@leiweibau - Looks great! I'll pull in the new version and have a play. Thanks for the additions :)

@accessiblepixel
Copy link
Author

accessiblepixel commented Apr 5, 2024

@Tuxtom007

@accessiblepixel

I have the same constellation, Pialtert runs on a LXC in Proxmox, I use 12 VLANs in my network and have given the PiAlert-LXC an additional network card in the first 3 VLANs as a test (each also with a static IP in the VLAN via DHCP reservation). An arpscan on the LXC directly with the network and the Inetrface works perfectly, I get all clients displayed as the 3 VLAN respectively.

But no matter how I enter it in pialert.conf, I don't get any clients from the other VLANs. I have already tried all possible combinations from the documentation: IP network + interface + VLAN, then without VLAN, only IP network + VLAN, etc.

I used to run PiAlert in the same way some time ago, it was a real pain to get all VLANs scanned, but unfortunately I can't remember how I did it.

Yesterday I switched to another fork, let's see which is better, the WebUI at least looks different and there is also the option to get the client from the Unifi controller.

Translated with DeepL.com (free version)

I've just had an issue that seems similar to what yours was. It was because the devices became undefined for the VLAN interfaces within the /etc/network/interfaces file.

Readding to that file and restartng the LXC got my arp scans working for the VLANs again.

...
auto eth5
iface eth5 inet manual
auto eth10
iface eth10 inet manual

Hope that helps!

@Tuxtom007
Copy link

@Tuxtom007
I've just had an issue that seems similar to what yours was. It was because the devices became undefined for the VLAN interfaces within the /etc/network/interfaces file.

Readding to that file and restartng the LXC got my arp scans working for the VLANs again.

...
auto eth5
iface eth5 inet manual
auto eth10
iface eth10 inet manual

Hope that helps!

Thanks.
I have found a solution that has been running for some time without any problems:

  • one NIC per VLAN in the LXC
  • only the main NIC has an IP, the other VLANs have no IP address

pialert.conf:

SCAN_SUBNETS = ['--localnet --interface=eth0','10.0.10.0/24 --interface=eth0.10 --arpspa=10.0.10.253','10.0.20.0/24 --interface=eth0.20 --arpspa=10.0.20.253','aso.....]

@accessiblepixel
Copy link
Author

@Tuxtom007 - yep that's exactly what I'm doing :> Glad you got it working.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants