[20.8.0] Unexpected interrupted return code #335
Comments
|
Hi @tuxmaster5000, could you provide please some more information about the scanner configuration? Is it a full & fast scan config? Some special option for scanner? does the target has something special (behind a firewall)? Did you see in the report or in the log any plugin timeout error / closed port ? |
|
No I don't use any special options, it is an simple full & fast scan. I only use the default setting for the scanner. The only noticeable is, that it will happens not on all hosts. An really error is not show in the report under the error page.
When I let run openvas in debug mode, I only see the typical error/warning messages about "Not launching because a mandatory key is missing (this is not an error)" on some tests. But the string "Scan process failure" don't occur in the debug log of openvas. Let an grep run over the debug log don't shown any real error.
When an real error will occur, I think openvas will log it in the debug mode. |
|
I am experiencing something similar with ospdf-openvas. Looks like the same error as above. I don't see how the error below is related to a configuration error. I havent looked into this in detail but set_scan_total_hosts looks like it has missing method being called from somewhere which I don't think is related to configuration errors but source code issues. OSPD[30973] 2020-10-20 08:30:06,334: ERROR: (ospd.ospd) While scanning: 164a83fe-17e9-42f9-b502-081e3c5b96b4 |
|
Same issue here as well. Details are as follows:
The report shows some NVT timeouts followed by "Scan Process Failure" and "Task interrupted unexpectedly", but the openvas.log output says:
No indication of errors in the log. |
|
I managed to fix my error by patching the source code in ospd_openvas/daemon.py replacing the except statement on line 1124 with " except:". |
|
@rolf-d2i , can you post an patch for it? Because on line 1124 I don't see an except statement: |
|
The issue i found is related to release 20.8.1 code, processing HOSTS_COUNT (starting on line 1119 in daemon.py), which doesn't work in that branch and version. Ignoring errors from this part of the code was good enough for me, it is more of a hack than a patch. |
|
The The patch from @rolf-d2i is wrong. It wont work and just resulting in a different issue. |
|
No this is wrong @bjoernrick. I was building OpenVAS in a docker container (see below). The 20.08 is supposed to be stable released version (when looking at the documentation and github), if it is not a stable released version then the clone command is picking up the wrong version of OpenVAS for some reason. |
|
Hi @rolf-d2i , |
|
As I mentioned before @jjnicola . I have built the version from source using what is documented as the latest stable version and encountering "Unexpected interrupted return issues". If there is an issue with the latest stable version then there should be at least a new issue when building the latest stable version. I was building everything from scratch using the same version. |
|
@rolf-d2i again you are mixing things up. A release can be found at the releases page of a repository at GitHub e.g. https://github.com/greenbone/ospd-openvas/releases Each release refers to a git tag. If you want to use the latest release you have to use https://github.com/greenbone/ospd-openvas/tree/v20.8.0 for ospd-openvas and https://github.com/greenbone/ospd/tree/v20.8.1 for ospd. New bugfix releases will be created from the release branches e.g. from https://github.com/greenbone/ospd/tree/ospd-20.08 If you are using a release branch of ospd-openvas you must use the release branch of ospd too. You can't combine the 20.8.1 release of ospd with the ospd-openvas 20.8 branch. The lastest stable version is not contained in a release branch. The release branch contains the next going to be released version. |
|
And also again your issue has nothing to do with the original reported one here. |
|
@bjoernricks as I mentioned before what versions should be used is poorly documented. Certainly the page "https://community.greenbone.net/t/gvm-20-08-stable-initial-release-2020-08-12/6312" doesn't clearly say anything about 20.08.X versions. It is unexpected behaviour that 20.08.01 release breaks with 20.08.00, you would commonly expect major.minor.patch as version numbering. The page lists for instance OSPd 20.8.1 together with 20.8.0 components implying you should use the latest 20.08.X version. This is honestly a very confusing version handling scheme/git management policy you are using. |
|
Yes this might be confusing from the outside and this specific error will not happen very often. But to be clear even if we would use semantic versioning (which we don't do) it wouldn't indicate that the unreleased ospd-openvas need an unreleased version of ospd or that a 21.1.0 versions of ospd-openvas doesn't work with 20.11.4. This is a task for the dependency management. The dependencies are set correctly for poetry but there seems to be a missing change in the setup.py file. In most situations it should be fine to mix the latest releases with using release branches. But personally I would never do that and I would not advise to do that. I don't even advise users to build from git checkouts. Even the release announcement says
So my general advise is to use the latest released versions. If you are familiar with fixing issues by yourself especially with debugging error messages like the one you got it is fine to build from git. |
|
Looks like this issue occurs after the scan itself has finished. For some reason the process stays hanging while status is Done/Finished and progress is 100%. I get this error after the NMAP scan and NASL tests are done. |
|
@tuxmaster5000 @CipherMonger Could you share if you set <exclude_hosts /> or share your create target XML? |
|
I repeatedly started scans with the following results: A fail returns: |
|
Yes, I have this set. Here is that section of my target XML: In my case, I get Interrupted at 99%. |
Could you try a scan where you only exclude 1 host/ip, instead of 2 and see if that works? |
|
Hi @zenire @CipherMonger Despite this ends in the same "interrupted scan", it seems not to be the same issue reported by @tuxmaster5000. IIUC, the original issue is a F&F scan with one host. Therefore, not possible a non scanned host for exclusion, non-resolved, etc. |
|
@jjnicola, Yes in my case the scan not stops at 99% or so. It will stops randomly between 39% and 69%. |
|
same issue, it would be better if gvm provider more detailed error msg instead of just saying stopped, here is the screenshots |
|
same |
|
I have the same problem:
And when are the 31+100+91+36+221+167 commits in the 20.08-branches released as stable? |
|
I also have this problem. Happens, randomly, on various different
tasks.
gvm@ov-master-eqi:~$ gvmd --versionGreenbone Vulnerability Manager
20.08.0~git-3276b8ff-gvmd-20.08GIT revision 3276b8ff-gvmd-20.08Manager
DB revision 233
gvm@ov-master-eqi:~$ uname -aLinux ov-master-eqi 5.0.0-32-generic
#34~18.04.2-Ubuntu SMP Thu Oct 10 10:36:02 UTC 2019 x86_64 x86_64
x86_64 GNU/Linux
…On Fri, 2021-01-08 at 08:38 -0800, Virsacer wrote:
I have the same problem:
Interrupted at 76 %
20.8.0 Release
==> /gvm/var/log/gvm/ospd-openvas.log <==OSPD[48] 2021-01-08
00:30:14,317: INFO: (ospd.command.command) Scan 6e9d313f-ab6e-44a7-
91c0-0c0e8a166134 added to the queue in position 1.OSPD[48] 2021-01-
08 00:30:15,178: INFO: (ospd.ospd) Currently 1 queued scans.OSPD[48]
2021-01-08 00:30:15,231: INFO: (ospd.ospd) Starting scan 6e9d313f-
ab6e-44a7-91c0-0c0e8a166134.
==> /gvm/var/log/gvm/gvmd.log <==event task:MESSAGE:2021-01-08
00h30.19 CET:2318: Status of task xxxxx (1efb9532-bda3-4935-9773-
319ba47f0417) has changed to RunningWARNING: cipher_setiv: ivlen=15
blklen=16
==> /gvm/var/log/gvm/ospd-openvas.log <==OSPD[48] 2021-01-08
02:11:29,582: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134:
Host scan finished.OSPD[48] 2021-01-08 02:11:29,584: INFO:
(ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan interrupted.
==> /gvm/var/log/gvm/gvmd.log <==event task:MESSAGE:2021-01-08
02h11.30 CET:2318: Status of task xxxxx (1efb9532-bda3-4935-9773-
319ba47f0417) has changed to Interrupted
==> /gvm/var/log/gvm/ospd-openvas.log <==OSPD[48] 2021-01-08
02:11:30,300: INFO: (ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134:
Scan stopped with errors.OSPD[48] 2021-01-08 02:11:30,301: INFO:
(ospd.ospd) 6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan
interrupted.OSPD[48] 2021-01-08 02:11:30,404: INFO: (ospd.ospd)
6e9d313f-ab6e-44a7-91c0-0c0e8a166134: Scan stopped with
errors.OSPD[48] 2021-01-08 02:11:30,404: INFO: (ospd.ospd) 6e9d313f-
ab6e-44a7-91c0-0c0e8a166134: Scan interrupted.
> In most situations it should be fine to mix the latest releases
> with using release branches. But personally I would never do that
> and I would not advise to do that. I don't even advise users to
> build from git checkouts. Even the release announcement says
> > GVM is published as regularly updated and tested source code
> > releases.
>
> So my general advise is to use the latest released versions. If you
> are familiar with fixing issues by yourself especially with
> debugging error messages like the one you got it is fine to build
> from git.
And when are the 31+100+91+36+221+167 commits in the 20.08-branches
released as stable?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
Hello, |
|
Note also that in my case; although the task is set as finished on openvas side; If I resume the task after Interruption it will resume where it left before crashing (67%), and after some time will be set to Interrupted state again. Restarting the task from scratch doesn't seems to help. |
|
Hi! Again thank you for your participation in hunting this bug! |
|
Hello,
Yes, I confirm I can reproduce the problem on one particular task, from
one particular scanner. I have different slave scanners running, and
one in particular is having this problem. Others never (or very barely)
suffer this issue. So, If I run a particular task from a particular
slave scanner, I reproduce the issue systematically (at least; my 3
latest tentatives in that scenario always ended in interrupted state at
the same percentage).
I am using the following version:
***@***.***:~$ /opt/gvm/bin/ospd-scanner/bin/ospd-openvas --version
OSP Server for openvas: 21.4.1OSP: 21.4.1OSPd OpenVAS: 21.4.1
***@***.***:~$ openvas --version
OpenVAS 21.4.1~git-51dcbd2f-
openvas-21.04GIT revision ~git-51dcbd2f-openvas-21.04gvm-libs
21.4.1~git-714a5ea2-gvm-libs-21.04
I'll install the latest 21.04 stable version and try again on the same
target. Hopefully the new error message will give us more informations.
Thanks !
…On Tue, 2021-06-15 at 06:51 -0700, Christoph Krämer wrote:
Hi!
Thank you for your report on this topic. I was working on this for
the past week and as before no one in my team was able to reproduce
it. What bothers me is that the times in the logs that you posted
does not match.
You said you are able to reproduce the issue systematically, does
that mean on the one specific task you always run into the same error
with an interrupted scan?
I am not quite sure: which version of ospd-openvas and openvas did
you use?
I added a log message to narrow down a specific case in which the
scanner could possibly get set as interrupted even so openvas is not
finished. This is in the newest versions of 20.08, 21.04 and master
branch.
Again thank you for your participation in hunting this bug!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub, or unsubscribe.
|
|
I've installed latest 21.04 and currently re-running the failing task with it. I'll let you know.* |
|
I've installed latest 21.04 and currently re-running the failing task with it. I'll let you know.* gvm@ov-slave-monterrey: gvm@ov-slave-monterrey:~$ /opt/gvm/bin/ospd-scanner/bin/ospd-openvas --version |
|
This is weird. The problem has gone on this particular task ! So far it happened systematically but the last scan done with the patched version worked without problem and was super fast. The only thing I did was update the OS to latest patches and reboot. I'm thinking it's perhaps a problem which occurs only with certain targets & probes. Nevertheless, I will roll out this version to all my slaves. We are running dozens of scans per week, so one time or another the same problem will arise again and I'll be able to report more debug informations. |
|
Yes this issue is really really weird. We still are not able to reproduce it reliably. Therefore we are very thankful for every feedback |
|
OK, I was able to reproduce the problem with your patched version. See below the latest logs just before task was set to Interrupted. Hope that helps. ospd Logs:
openvas.log
This is weird, as both ospd and openvas confirm the scan task is finished, but instead of putting a 100% score and settings the task as finished, it immediately put it as Interrupted without further details. I don't know what the patch was suppose to do, but it seems to me it failed since I see no difference in the logs vs previous unpatched version. |
|
I have an idea for it. I have seen this messages last week, when no redis connection was available any more. Because to much parallel jobs was running. After allowing redis more parallel databases it was gone. I have set databases to 512 in the redis.conf |
|
@tuxmaster5000 not my case here unfortunately. My redis databases number here is set to 65535... |
|
I have one job sheduled every day and 4 jobs spread over the weekends. With 20.8.0 and 21.4.0 the daily job did complete about every ten days or so. On thursday I updated to the latest from 21.04-branch and all jobs have completed successfully since then :) |
|
@Virsacer that's the problem. This issue is completely erratic. Sometimes it happens all the times, sometimes everything runs without issues. Right now I have two tasks which constantly ends up in interrupted state; while using the latest 21.04 version. But I don't see anything in the logs explaining why a running task, right after being set as "finished" by ospd is put in Interrupted state. That's completely weird. |
|
To give you a short update: I put quite some work into restructuring the code and eliminating some possible race conditions. These changes are contained in the master and 21.04 branch. Thanks again for the many feedback we get regarding this topic! |
|
Thanks Kraemii. Will give it a try and let you know. Regards |
|
HI everyone! I was able to reproduce the issue for a few scans, even with the patch from @Kraemii, since the problem was in openvas-scanner. This got me some hints. The issue should be fixed with greenbone/openvas-scanner#832 . |
|
Thanks guys for the update and trying to fix this problem. I've applied the patch from Kraemii a couple of weeks ago, and ran dozens of scans after that on targets usually prone to this problem, and so far I wasn't able to reproduce the issue anymore. So even if it doesn't fix totally the problem, it seems to help. I'll continue to report any new findings here. |
|
All, I had the problem again, but now the situation is a bit different. Basically what happens now is:
`event task:MESSAGE:2021-08-11 08h11.21 UTC:1788: Status of task Cedar Rapids, VOIP (c54c7ddb-a119-4417-8bbd-7952f6f2faab) has changed to Stop Requested ` While on ospd side, you have:
Step 2) should ring some bells to old timers. We exactly had this very problem with GVM 11, and this bug was closed as too old. But seems it hasn't been resolved either... There is really something broken in the communication protocol between ospd and gvmd. How comes gvmd has so many issues tracking scan progress & status, while everything seems to work quite fine on ospd side ? On my end it seems I can narrow down the problem to one scanner on some tasks in particular. I'llupdate ospd/openvas with #882 and see if it happens again. |
|
Same problem with patch #832. Scans are continuing to run on ospd side; while frozen then stopped on gvmd side. Trying to resume the task will actually:
|
|
Hi @wisukind , Thanks for reporting. |
|
Yes, it's indeed a different issue. Here ospd / openvas are working fine. On gvmd side I had several connection lost logs lines when the issue occured; so that's fine. But my question is mostly on the resuming part; why gvmd start a new scan when the connection is back, instead of resuming the already finished one (and putting the state as Done) ? And why gvmd puts immediately the resumed task as Interrupted ? Overrall that's always the same story; gvmd is not reliably maintaining the communication with ospd and loose tracks of scans tasks too easily. It should be able to resume tasks where it left, and also to get up to date scan status from the scanner directly, even after the connection has dropped for some time. Should I open a new bug report for this ? |
|
Yes, what you are experimenting now is a complete different issue. Feel free to open a new issue for gvmd. Regarding the current issue, despite I think the main issue here (ospd putting the scanner as interrupted) is already addressed, I will keep it open some more time, waiting for more feedback about openvas/ospd-openvas behaviour. |
|
@wisukind And as always, thank you very much for your feedback! ;) |
|
Yes, please leave that bug open for now. I am running dozens of scans per month, so this is usually happening very frequently. If it doesn't happen again by, let's say, mid september, then you should be fine to close it. Also I've created bug #1668 on gvmd side. |
|
Hello, After a few weeks of testing on targets prone to this bug, I confirm the bug has just moved to issue #1668. Basically, tasks are no longer put in Interrupted state, but in Stopped state now for no reasons. On some of my tasks, it's systemic. It turns out, however, that the bug is now on gvmd side, as on ospd the task appears as finished. |
|
I am glad to read that last comment and can closed this issue. Thanks a lot to those who have tested the the different patches and given feedback. So, closing here. |
Use empty string instead of None for credentials value
|
Hello. This bug should unfortunately be re-opened. I am witnessing that problem again on some tasks. Highly less frequent, but still happens in 21.4.3 final release. :-( |
|
Hello @wisukind,
Please also set the log level to debug for OSPD and OpenVAS and add some logs if possible. Please. Open a new issue. Thanks in advanced. |
The scanner self will exits fine, but ospd-openvas will interpret it as an error.
ospd-openvas log:
openvas log:
No error was logged by openvas.
openvas --version
ospd version:
Most new code since 2005: (C) 2020 Greenbone Networks GmbH
Nessus origin: (C) 2004 Renaud Deraison deraison@nessus.org
License GPLv2: GNU GPL version 2
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
The text was updated successfully, but these errors were encountered: