Import fails on large property_name (>64)

[joni1993]

Expected Behavior

Import should be successfull even on larger property_names

Current Behavior

Import fails with error complaining about property_name being to large.

Possible Solution

See #1712

Steps to Reproduce (for bugs)

1. Install the icingaweb2-module-netboximport module, which import Objects directly from netbox, which uses large names like "local_context_data__last_inform__port_table__3__mac_table__0__uptime"
   Alternatively it should also be triggered without the netboximport module when using large names in the property_name.
2. Setup Import module
3. Try to import
4. Import Fails with "Data too long for column 'property_name'"

Your Environment

• Director version (System - About): master (e895978)
• Icinga Web 2 version and modules (System - About): 2.6.1, icingaweb2-module-netboximport
• Icinga 2 version (icinga2 --version): r2.9.2-1
• Operating System and version: Ubuntu 16.04
• Webserver, PHP versions: Apache (behind nginx reverse), PHP7.2

[Thomas-Gelf]

@joni1993: I'm not so happy with this change, as it has a notable performance impact and no real-world value. local_context_data__last_inform__port_table__3__mac_table__0__uptime isn't something I would consider being very helpful as a property, and uptime is a volatile property that shouldn't be imported at all as it changes constantly. I'd suggest to address this problem in the related module.

[joni1993]

I dont know the Database structure very well but why is the table "import_row_modifier" using "property_name VARCHAR(255) NOT NULL," then but "imported_property" is only using VARCHAR(64)?

[Thomas-Gelf]

You're right, it should be VARCHAR(64) also in import_row_modifier. imported_property contains MUCH more data than import_row_modifier, so it that table it doesn't hurt as much as it would in the former one.

I had a quick look at the netboximport module, do you know why it flattens data at all? There is no reason for doing so, Import Sources are well allowed to ship structured data.

[joni1993]

No i don't know why it flattens it but probably because of easier access to variables.
I just removing the flattening in the module and i got for example a property "primary_ipv4" with the contents:

{
  address: "10.4.0.2/24",
  family: 4,
  id: 25,
  url: "http://***/api/ipam/ip-addresses/25/"
}

To access the address i tried apply a modification to the property with the "Decode a JSON string" function but the result is always null. Is the "Decode a JSON string" here wrong?

[Thomas-Gelf]

No need to decode this, if the module provides raw data to the Director this is internally already handled as structured data. If the shown data is the value for the primary_ipv4 column you can use primary_ipv4.address to reach that field. If the module doesn't ship these nested names in it's listColumns() method then please use Custom Expression -> ${primary_ipv4.address} in your related Sync Rule property.

[joni1993]

Thanks, I didn't know that. Would be great the have that in the documentation.
Altough i get an Error on Sync:

fehl: Data is not nested, cannot access primary_ipv4.address: stdClass::__set_state(array( 'name' => 'HOSTNAME', 'primary_ipv4' => NULL, 'display_name' => NULL, 'primary_ipv6' => NULL, ))

Is this due the fact that the source is delivering the address in the format: "1.2.3.4/24"?
Maybe this is one reason of the flattening, because you cant apply modifications on JSON data that way?

Another thing is that i cant use a filter expression like ${custom_fields.Monitoring}=True in the Sync rule. Thats probably another reason for flattening.

[Thomas-Gelf]

Could you please show me an (anonymized) preview screenshot for your import source?

[joni1993]

Edit: The failure is probably the stripped quotes on the JSON properties.

[Thomas-Gelf]

Oh, now I see -> in the former error message it shows:

... 'name' => 'HOSTNAME', 'primary_ipv4' => NULL, ...

The first row seems to be the problem, does the module ship a Title row or similar as a first row? You could try to use the Black- or Whitelist... Property Modifier to get rid of such invalid rows.

[joni1993]

No i dont think it does so. But i think its maybe an error in the JSON output because the Properties are not in double quotes.

I think its a problem with the Module in the first place so i dont want to waste your time any longer.

Thanks for your support and your awesome product!

[Thomas-Gelf]

You're welcome! If you (or the module writer) need more help please let me know.

[Thomas-Gelf]

I'll close this issue for now, as I'd prefer to not increase the property_name field length.