New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Custom vars update via API not working #6576

Closed
routetonull opened this Issue Aug 26, 2018 · 7 comments

Comments

Projects
None yet
3 participants
@routetonull

routetonull commented Aug 26, 2018

System information:

Icinga2 version: r2.9.1-1
Icinga Web 2 Version 2.6.1

System information:
Platform: Ubuntu
Platform version: 18.04.1 LTS (Bionic Beaver)
Kernel: Linux
Kernel version: 4.15.0-29-generic
Architecture: x86_64

Problem description

When updating host object vars via API the API shows the variables but they are not reflected in icinga configuration and not used for host group assignment.

How the issue was detected

Host "test1" is created via API

curl -k -s -u $ICINGAUSER:$ICINGAPASSWORD -H 'Accept: application/json' -X PUT 'https://10.16.100.
16:5665/v1/objects/hosts/test1' \
> -d '{ "templates": [ "generic-host" ], "attrs": { "address": "192.168.1.1"}, "pretty": true }'
{
    "results": [
        {
            "code": 200.0,
            "status": "Object was created"
        }
    ]
}

Open the API page of the host, vars is null, that's ok.

Icinga CLI doesn't show test1 but it exists in web gui

icinga2 object list --type=host --name=test1

Restart icinga2 service, object test1 is created, run again the same command and focus on vars:

icinga2 object list --type=host --name=test1 | grep -A2 vars
  * vars = null
  * volatile = false
  * zone = "icinga"

Create an host group matching vars.role = switch. This is where the problem will happen.

object HostGroup "switch" {
  display_name = "switch"

  assign where "switch" in host.vars.role
}

Now update object test1 with custom variable vars.role with value ["switch"] it's an array:

curl -k -s -u $ICINGAUSER:$ICINGAPASSWORD -H 'Accept: application/json' -X POST 'https://10.16.100.16:5665/v1/objects/hosts/test1' \
-d '{ "attrs": {  "vars.role" : ["switch"] }, "pretty": true, "verbose" : 1 }'

The new variable and its value are shown in the web gui and also in the API link

https://10.16.100.16:5665/v1/objects/hosts/test1

Image: https://i.imgur.com/cUHPxVm.png

but the CLI command still shows vars = null

icinga2 object list --type=host --name=test1 | grep -A2 vars
  * vars = null
  * volatile = false
  * zone = "icinga"

Host test1 is not assigned to group switch.

Now create another object testOK but this time add the custom variable when creating the object

curl -k -s -u $ICINGAUSER:$ICINGAPASSWORD -H 'Accept: application/json' -X PUT 'https://10.16.100.16:5665/v1/objects/hosts/testOK' \
-d '{ "templates": [ "generic-host" ], "attrs": { "address": "192.168.1.2","vars.role" : ["switch"]}, "pretty": true }'

The gui shows the very same output of the custom variable.

Image: https://i.imgur.com/0R4Hl5x.png

Accessing the API via browser shows the same output for custom variable

https://10.16.100.16:5665/v1/objects/hosts/testOK

Image: https://i.imgur.com/cUHPxVm.png

the host is member of the host group:

Image: https://i.imgur.com/nUZFChL.png

The icinga CLI still doesn't show the host:

icinga2 object list --type=host --name=testOK

Reload the service and try again. The host now is there and the vars have the correct value:

icinga2 object list --type=host --name=testOK | grep -A2 vars
  * vars
    * role = [ "switch" ]
      % = modified in '/var/lib/icinga2/api/packages/_api/4d074d06-3b0b-42d8-9616-3cff1d03c93d/conf.d/hosts/testOK.conf', lines 5:2-5:28

Try again with objet test1 to see the difference:

icinga2 object list --type=host --name=test1 | grep -A2 vars
  * vars = null
  * volatile = false
  * zone = "icinga"

Results

Create a new object then add variable:

  • GUI show variable OK
  • API shows variable OK
  • host is NOT matching the assignment to host group
  • icinga2 object list shows variable = null

Create new object with variable in the same POST operation

  • GUI show variable OK
  • API shows variable OK
  • host is matching the assignment to host group OK
  • icinga2 object list shows variable with the expected value OK

Conclusion

When updating the object via API something is lost in the process. Or maybe I'm doing something wrong.

Any ideas? Thanks!

@Crunsher

This comment has been minimized.

Member

Crunsher commented Aug 27, 2018

icinga2 object list does not show runtime modifications, neither are group memberships re-applied on runtime changes.

Does it work as expected when you reload icinga?

@Crunsher

This comment has been minimized.

Member

Crunsher commented Aug 27, 2018

I'm sorry I missed this

Restart icinga2 service, object test1 is created, run again the same command and focus on vars:

This may be related to #6569 then

@routetonull

This comment has been minimized.

routetonull commented Aug 27, 2018

I think you're right, it seems the same issue of #6569

If any other log/dump can help to fix the issue I can reproduce it in my environment.

@Crunsher

This comment has been minimized.

Member

Crunsher commented Aug 28, 2018

A dump of icinga_customvariables and icinga_customvariablestate from the ido for a few of the affected vars would be welcome. Also does this only happen to hosts? If yes the config of one such host, at best it uses the custom var from the sql dump.

Once you got that you can try this workaround, it should work and if it does not that's also information we can use.

@Crunsher Crunsher added the duplicate label Aug 28, 2018

@routetonull

This comment has been minimized.

routetonull commented Aug 29, 2018

All the hosts configs are pushed through API. To get the config file I use

icinga2 object list --type=host --name=HOSTNAME
It validates the assumption the vars are not pushed in the config file with POST operations.

This host was created via API but vars where pushed during object creation:

object Host "testOK" {
import "generic-host"

    address = "192.168.1.2"
    vars["role"] = [ "switch" ]
    version = 1535301085.004249
    zone = "icinga"

}

This host was created without any custom var, the var was added in another POST operation wia API:

object Host "wlc6" {
        import "generic-host"

        address = "10.1.1.1"
        version = 1535279899.968291
        zone = "icinga"
}

Custom variables are present in in tables icinga_customvariables and icinga_customvariablestatus.

mysql> select * from icinga_customvariables where varname = 'role';
+-------------------+-------------+-----------+-------------+-------------------+---------+------------+---------+
| customvariable_id | instance_id | object_id | config_type | has_been_modified | varname | varvalue   | is_json |
+-------------------+-------------+-----------+-------------+-------------------+---------+------------+---------+
|              1262 |           1 |       722 |           1 |                 0 | role    | ["switch"] |       1 |
|              1280 |           1 |       726 |           1 |                 0 | role    | ["switch"] |       1 |
|              1281 |           1 |       730 |           1 |                 0 | role    | ["switch"] |       1 |
|              1282 |           1 |       720 |           1 |                 0 | role    | switch     |       0 |
+-------------------+-------------+-----------+-------------+-------------------+---------+------------+---------+

icinga_customvariablestatus

mysql> select * from icinga_customvariablestatus where varname = 'role';
+-------------------------+-------------+-----------+---------------------+-------------------+---------+------------+---------+--------------------+
| customvariablestatus_id | instance_id | object_id | status_update_time  | has_been_modified | varname | varvalue   | is_json | endpoint_object_id |
+-------------------------+-------------+-----------+---------------------+-------------------+---------+------------+---------+--------------------+
|                    1269 |           1 |       722 | 2018-08-26 17:32:19 |                 0 | role    | ["switch"] |       1 |               NULL |
|                    1291 |           1 |       726 | 2018-08-29 16:48:15 |                 0 | role    | ["switch"] |       1 |               NULL |
|                    1292 |           1 |       730 | 2018-08-29 16:48:15 |                 0 | role    | ["switch"] |       1 |               NULL |
|                    1293 |           1 |       720 | 2018-08-29 16:48:15 |                 0 | role    | switch     |       0 |               NULL |
+-------------------------+-------------+-----------+---------------------+-------------------+---------+------------+---------+--------------------+

Suggested workaround doesn't work:

UPDATE icinga_hosts SET config_hash = NULL;

@Crunsher

This comment has been minimized.

Member

Crunsher commented Sep 3, 2018

Sorry for the late reply, the fact that the workaround does not work is interesting, that means the issue is not related to the checksums as suspected but with the IDO writer itself. The tables look normal but I will try to reproduce this with vars created via the API.

Thank you very much for helping us out 🙇‍♀️

@dnsmichi dnsmichi removed the duplicate label Sep 6, 2018

@dnsmichi

This comment has been minimized.

Member

dnsmichi commented Sep 6, 2018

Summary.

  1. Objects created or modified at runtime are not immediately written to the icinga2.debug file which acts as cache for icinga2 object list. Therefore you'll need to do a reload/restart to fully reflect runtime changes towards object list. This has been added to the docs with #5140.

  2. Apply rules and group assign statesments are not re-evaluated "live" when an object has been created or modified. In your case, the hostgroup assign won't be triggered again by the custom attribute modification. You need to reload/restart Icinga to evaluate the expression again. There's a tracking issue with #4084, unlikely to be implemented soon.

  3. CV workaround is tracked with #6569.

@dnsmichi dnsmichi closed this Sep 6, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment