New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Devices still show in lists despite being deleted #2937
Comments
Are you running this on the poller that normally collects from the devices? |
I actually ran it from the main poller first which showed the devices in the DB then ran it on the remote poller that time it didnt show the devices in the db |
I believe that now when a device gets deleted in Cacti, it is flagged as deleted using the deleted column and then this is replicated. This was a change in 1.2 to ensure that deleted devices were properly replicated for deletion rather than simply removing the record only to have it recreated because the main or remote thought it should be there. I'm wondering if there is a conflict between this deleted flag being set and the record not removed yet, and automation checking if a device exists without filtering on the deleted column. |
its weird I did a full sync as well and that didn't seem to help |
The only time that this could possibly happens is when:
Case 1) is difficult to completely overcome. We do have a purge command that happens at the bottom of each poller run, but if the remote database is not accessible, the purge will not complete successfully. |
It's got me scratching my head because even the remove device php script
comes back as successful
I ran the script on both the main and remote pollers as well
The main poller found all of the devices that I was looking for I ran with
a --confirm result came back as successful
The devices did not show up in the device group and red cleaner marked the
rrrds as orphaned
…On Sun., Sep. 15, 2019, 10:38 a.m. Jimmy Conner, ***@***.***> wrote:
The only time that this could possibly happens is when:
1. When devices are removed while the remote poller is running
2. When the main data collector can not reach the remote database over
a mysql connection.
Case 1) is difficult to completely overcome. We do have a purge command
that happens at the bottom of each poller run, but if the remote database
is not accessible, the purge will not complete successfully.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2937?email_source=notifications&email_token=ADGEXTGKGXH3ONNN4BP5JK3QJZCFDA5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XSA5Y#issuecomment-531570807>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGEXTAXY7LEVSRQ476JW4TQJZCFDANCNFSM4IWJVQAQ>
.
|
From the main system, edit the Remote Data Collector and test the connection. Is everything coming back successful? |
I'll check that tomorrow and let you know
Thanks!
…On Sun., Sep. 15, 2019, 10:58 a.m. Jimmy Conner, ***@***.***> wrote:
From the main system, edit the Remote Data Collector and test the
connection. Is everything coming back successful?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2937?email_source=notifications&email_token=ADGEXTH4LEOX2HK2XN74MJDQJZERLA5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XSPBA#issuecomment-531572612>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGEXTFWFC2PAGSWTKSULWTQJZERLANCNFSM4IWJVQAQ>
.
|
Just a thought though
If the remote pollers couldn't access the main poller wouldn't I have
bigger issues
The graphs from that poller work just fine and that poller gets all the
settings properly even when updated at the main poller
Also there are no errors with poller syncs
On Sun., Sep. 15, 2019, 11:42 a.m. Sean Mancini, <sean@seanmancini.com>
wrote:
… I'll check that tomorrow and let you know
Thanks!
On Sun., Sep. 15, 2019, 10:58 a.m. Jimmy Conner, ***@***.***>
wrote:
> From the main system, edit the Remote Data Collector and test the
> connection. Is everything coming back successful?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <#2937?email_source=notifications&email_token=ADGEXTH4LEOX2HK2XN74MJDQJZERLA5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD6XSPBA#issuecomment-531572612>,
> or mute the thread
> <https://github.com/notifications/unsubscribe-auth/ADGEXTFWFC2PAGSWTKSULWTQJZERLANCNFSM4IWJVQAQ>
> .
>
|
Ok tested The main poller is able to reach the remote pollers DB just fine Still very weird |
Ok so I revisited this and something is still no right When you delete the device off the main poller it is gone you can add the site back manually but through AUTOM8 is will show as still in cacti If you select the hostname table the entry is still in the DB ! The main poller can reach the remote pollers just fine replication works as well |
That is likely that AUTOM8 isn't ignoring devices that are set to deleted in the hosts table. Because of replication, they are not totally removed until all replication is complete. |
I forced replication and still the same
It's like the table is not actually replicating
…On Sun., Oct. 6, 2019, 12:32 p.m. Mark Brugnoli-Vinten, < ***@***.***> wrote:
That is likely that AUTOM8 isn't ignoring devices that are set to deleted
in the hosts table. Because of replication, they are not totally removed
until all replication is complete.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2937?email_source=notifications&email_token=ADGEXTCO2TQIIS33CVX2DDLQNIHIHA5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEAOOFFI#issuecomment-538763925>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGEXTGRKC7SOSRAGWHRWFTQNIHIHANCNFSM4IWJVQAQ>
.
|
Deleted a device directly from the DB here is the command I ran (delete from host where hostname like 'Ip address;) I re-ran autom8 and it picked up the device and re-added it No errors are seen from the SQL call and no troubles issuing the command as the same user that cacti uses so not a permissions issue I am not sure if this is specific to 1.2.5 |
Deleting the device manually will leave behind datasources and graphs related to it. It will also not sync to the other pollers. You should make sure that you don't have rows in other tables (especially tables with a host_id) that match. If you do, you'll have to start cascading through the tables or you could end up with other issues were code starts at a data source or graph and works backwards. |
I would do a select on the device you want to delete manually before you do to keep a record of it. |
Thanks
I'm not planning on doing this regularly because cacti should be deleting
devices properly
I just wanted to be sure all permission were fine
Still need to figure out why cacti is not doing this properly
…On Tue., Oct. 8, 2019, 3:29 a.m. Mark Brugnoli-Vinten, < ***@***.***> wrote:
I would do a select on the device you want to delete manually before you
do to keep a record of it.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2937?email_source=notifications&email_token=ADGEXTBFNRDOV7T7TSN36DLQNQZGVA5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEATFQHI#issuecomment-539383837>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/ADGEXTCD55UKB5QXJ75637TQNQZGVANCNFSM4IWJVQAQ>
.
|
This is still ongoing I have run the audit db script all good analyze ab all good I still see entries in the db from deleted devices I don't think the entries ate being marked for deletion |
Oh and this is on 1.2.7 |
You need to be on branch 1.2.x. Also, perform a diff between config.php.dist and config.php, let me know what is different. I really suspect that you are having some database communication problems. Are you sure you don't have a database issue? Test connection comes back good for all collectors? |
I will do the diff and get back to you
Everything works I can add devices even on remote pollers no issue
Clicking the test button works fine
Mysql from cli work both from remote to.main and main to remote
Poller replication works no errors reported
I would think if it was a db connect issue it would have manifested in
other ways though
…On Sat., Nov. 9, 2019, 7:05 p.m. Jimmy Conner, ***@***.***> wrote:
You need to be on branch 1.2.x. Also, perform a diff between
config.php.dist and config.php, let me know what is different. I really
suspect that you are having some database communication problems. Are you
sure you don't have a database issue? Test connection comes back good for
all collectors?
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2937?email_source=notifications&email_token=ADGEXTAJIOUMN2GNR3NPU2LQS5F4RA5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDUSACA#issuecomment-552149000>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADGEXTDXXUS6PBT5JH5EDSLQS5F4RANCNFSM4IWJVQAQ>
.
|
How large is the install? What's you lowest collection frequency? I'll do some testing now that I've installed a remote data collector. |
Ok so the install we have has 2100 devices
With 1 main and 2 remote pollers
1 minute polling interveral
I don't have the polling stats right now
…On Sat., Nov. 9, 2019, 7:12 p.m. Jimmy Conner, ***@***.***> wrote:
How large is the install? What's you lowest collection frequency? I'll do
some testing now that I've installed a remote data collector.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2937?email_source=notifications&email_token=ADGEXTCPZZT2DFUJKIDNGN3QS5GXRA5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEDUSEFA#issuecomment-552149524>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADGEXTDITJBVCAM74KTQNPDQS5GXRANCNFSM4IWJVQAQ>
.
|
Ok a fresh example multi pollers setup 1.2.7 not the same as the above examples I forced a sync between the pollers here is the output 2019/11/12 20:19:45 - WEBUI NOTE: All selected Remote Data Collectors in [2] synchronized correctly by user admin2019/11/12 20:19:45 - CMDPHP NOTE: Table data_input_data Not Replicated to Remote Poller 2 Due to No Rows Found the remote poller shows 2 devices assigned to it ??? Second poller device view See below for sql test [root@localhost ~]# mysql -u cacti -p -h 192.168.1.228 Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. MariaDB [(none)]> use cacti; Database changed MariaDB [cacti]> select hostname from host; |
Hey Guys I am still seeing this any new ideas ? |
Hey guys Wondering if you have had a chance to look at this any info I can provide please let me know ! |
so funny thing I found when I delete the device it does not show up in the device list but shows up everywhere else It shows up in the auto-complete dropdown when you search for it in the device view but if you click it nothing shows up In the monitor plugin tab the device shows in the device view please I am at a loss here any help would be appreciated |
Oh btw on the recent device deletion I forced a poller sync as well same outcome |
Yeah I am not sure I know that in 1.2.4 this was not an issue not quite sure when the deleted column was introduced when I was doing testing for autom8 we would delete and re-add devices a lot without issues I am sure that something in the background is not picking up the tag I belive its also happening on the main poller as well so I dont think its a replication issue as well but I will test again just to be sure |
I have a theory could this be THOLD holding on to the record? The device table in the DB does indeed have this set as a deleted device but why even after 50 days does it insist on staying ?? |
Hey Guys While we look into this would you be able to tell me of a manual way to clear these entries ? |
I found this block of code in api_device function api_device_purge_deleted_devices() { So this would mean that after 24 hours that the devices should be purged I belive something is wrong with this block since I have seen devices both in my production and multiple labs stick around for days When I run the api_device.php I see the below php api_device.php -h Here is line 25 of api_device.php include_once($config['base_path'] . '/lib/poller.php'); /* api_device_crc_update - update hash stored in settings table to inform
poller.php is for sure there ls /var/www/html/cacti/lib/poller.php -lah |
Run this, post the output: SELECT id, deleted, last_updated FROM host WHERE deleted="on"; In general, I don't know why there is a library include inside of lib/api_device.php. That is simply not right, especially if it's not within a function definition. |
* We should not be perfomring includes in the base of any library file.
I've removed the include in lib/api_device.php. But to apply it, you have to take all the files. It should be pretty harmless. |
Hey thanks!
I will try it do you mean to take all the files in lib?
…On Wed., Jan. 22, 2020, 8:19 a.m. South Park Neo, ***@***.***> wrote:
I've removed the include in lib/api_device.php. But to apply it, you have
to take all the files. It should be pretty harmless.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#2937?email_source=notifications&email_token=ADGEXTCRFVJHXSY2YAXMG4LQ7BBU7A5CNFSM4IWJVQA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJTQQMA#issuecomment-577177648>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ADGEXTCCRYSZB74ZZX2H7HLQ7BBU7ANCNFSM4IWJVQAQ>
.
|
Here is the output from the db SELECT id, deleted, last_updated FROM host WHERE deleted="on"; |
Well, that explain it then. Just need to find out where the update is occurring. |
I applied the new files autom8 is working fine no errors seen as of yet |
@TheWitness the update is happening at each poll |
Ok so I added a device via autom8 now and deleted it its been 5 mins and the last updated shows as of 5 mins ago so something is definitely different the odd thing is that the other devices still show updating currently |
| 4756 | on | 2020-01-22 10:36:05 | |
I'm in the process of reviewing code that makes updates to the |
* Devices still show in cacti despite being deleted * PHP Error when Creating New Graphs through Automatically Added Devices using Sync Device Template
Okay, think this is fixed now. There is a spine update too. Should be committed within a few minutes. |
Dude thanks so much I will definitely test this and get back to you ! |
Devices still show in cacti despite being deleted
So, I did some testing and found that discovery that was made on a remote data collector did not get pushed to the local databases, forcing a full sync to resolve that. The commit that I just made ensures that you don't have to full sync to make the discovery work. This this things goose is cooked now. I also tightened the screws on the purge timing to just under 10 minutes. |
Not sure where you are going with that? |
On the original issue, I am seeing that this seems to be fixed in 1.2.9 in my lab testing anyways |
Hey Guys
So I deleted about 100 Devices to re-inport them via automation into cacti
when I try to re-import cacti skips the IP's reporting device already in cacti
I tried the remove_device script and used a regex which found all the sites I used --confirm the script reports back successful
yet AutoM8 still skips the IP's and if you run the script again the devices are still there !!
The text was updated successfully, but these errors were encountered: