[Citrix ADC] Syslog messages are not according to documentation

[fgierlinger]

As @eriroley mentioned in #9592 , the Citrix ADC messages received over syslog are missing the fields

• syslog timestamp
• level
• client ip

even though the documentation of citrix adc clearly states, that the logs are to be expected in this format. Therefore the grok pattern fails. We double checked all the settings, but the log message format is not configurable for adc logs.

As of our understanding, the logfile format with the timestamp, level and client ip is only valid for logs saved to the /var/log folder.

integrations/packages/citrix_adc/data_stream/log/elasticsearch/ingest_pipeline/default.yml

Lines 26 to 37 in 7cdf580

	- grok:
	description: Extract header details and message from log line.
	tag: grok_event_original
	field: event.original
	patterns:
	- '^%{SYSLOG_TIMESTAMP}( %{LEVEL} %{IP:client.ip:ip})?( %{HOSTNAME:citrix.hostname})? %{GREEDYDATA:citrix.detail}'
	- '^%{GREEDYDATA:citrix.detail}'
	pattern_definitions:
	LEVEL: '<?%{IDENT:citrix.facility:keyword}\.%{IDENT:citrix.priority:keyword}>?'
	IDENT: '[a-zA-Z][a-zA-Z0-9]*'
	SYSLOG_TIMESTAMP: '(?:%{SYSLOGTIMESTAMP:_tmp.timestamp}|%{TIMESTAMP_ISO8601:_tmp.timestamp8601})'
	TIMESTAMP_ISO8601: '%{YEAR}-%{MONTHNUM}-%{MONTHDAY}[T ]%{HOUR}:?%{MINUTE}(?::?%{SECOND})?%{ISO8601_TIMEZONE:event.timezone}?'

Here are some example messages:

<134> 06/14/2024:18:29:23 GMT host1 0-PPE-1 : default REWRITE Message 209051714 0 :  "Source: 1.1.1.1 Destination: 192.168.0.1 Request URL: site.example.com/webcam/1.jpg HTTP Status: 200"
<134> 06/14/2024:16:02:01 GMT host1 0-PPE-0 : default TCP CONN_TERMINATE 225601598 0 :  Source 192.168.0.1:443 - Destination 192.168.0.2:15567 - Start Time 06/14/2024:16:02:01 GMT - End Time 06/14/2024:16:02:01 GMT - Total_bytes_send 0 - Total_bytes_recv 1
<134> 06/14/2024:16:01:40 GMT host1 0-PPE-0 : default TCP CONN_DELINK 220659229 0 :  Source 1.1.1.1:48277 - Vserver 10.0.0.1:443 - NatIP 192.168.0.1:12248 - Destination 192.168.0.100:443 - Delink Time 06/14/2024:16:01:40 GMT - Total_bytes_send 0 - Total_bytes_recv 619
<135> 06/14/2024:16:01:39 GMT host1 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_SUCCESS 220659228 0 :  SPCBId 27719517 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" - Session New  - HandshakeTime 120 ms

[jamiehynds]

@piyush-elastic could you please look at this one.

[eriroley]

I got an email notification that I'm not seeing in the thread here - but, to answer the question - yes, our ADC/Netscaler has been in production for quite a while (we keep it up-to-date), it is a virtual appliance that was migrated from a legacy MPX back in the day.

Are you thinking that there is a legacy setting on the ADC that needs to be adjusted?

[jvalente-salemstate]

I got an email notification that I'm not seeing in the thread here - but, to answer the question - yes, our ADC/Netscaler has been in production for quite a while (we keep it up-to-date), it is a virtual appliance that was migrated from a legacy MPX back in the day.

Are you thinking that there is a legacy setting on the ADC that needs to be adjusted?

that was me. I had a mistake in there but wasn't near a computer to correct it.

But for anyone who missed it, I'd asked the device was from before Citrix rebranded the Netscaler to ADC. The non-working format in the OP does match older parsers from before that, and our logs also have this format (device purchased prior to 2020, but up to date in terms of version).

I do think that it may be a scenario where the format isn't updated (maybe to not break existing logging ?) on older devices. I don't know how to verify or change that though as I'm not our Citrix admin, or I'd be able to test this before waiting for our new unit to be up.

[jvalente-salemstate]

So looking closer at our configuration, I think what we're seeing is the use of classic policies. These were deprecated before 13.0. The logs in /var/log/ appear as expected since the export policies don't apply to them. I am also going to guess using syslog.conf to specify an agent with the integration as the syslog server would work. If it is the classic policies, we should probably be moving off those 😀

However, I don't think that is the issue here. Looking at the pipeline, it does fail to match that first grok pattern but the entire event.original is sent to the native pipeline or CEF pipeline after being matched and stored in another field with the second pattern. There's a grok pattern in the Native pipeline that should be catching this format and I can't determine why it's not working. There's a space before the PRI in the pattern. Could that be it?

[jvalente-salemstate]

@fgierlinger @eriroley

Could either of you share a few more (redacted) sample logs or the error message?

I applied the following to our new, and old, devices

add audit syslogAction elastic_agent_vip x.x.x.x -serverPort 10610 -logLevel EMERGENCY ALERT CRITICAL ERROR WARNING NOTICE INFORMATIONAL -tcp ALL -acl ENABLED -userDefinedAuditlog YES -appflowExport ENABLED -lsn ENABLED -alg ENABLED -subscriberLog ENABLED -dns ENABLED -ContentInspectionLog ENABLED -sslInterception ENABLED -urlFiltering ENABLED -timeZone LOCAL_TIME
add audit syslogPolicy elastic_agent_syslog TRUE elastic_agent_vip
bind audit syslogGlobal -policyName elastic_agent_syslog -priority 10

The format still differs but in my case I am missing the time zone (need to try not setting it in the config I guess) but after I edited the native pipeline to make that an optional match it works.

But the sample logs in the OP also worked without any edits via the grok debugger.

edit: looking at logs that have come in, it does look like additional parsers for things like Citrix Gateway / Storefront events are needed.

[jvalente-salemstate]

Sorry for 4 comments here but I've identified the cause. There is/may be a trailing space at the end of event.original and the grok patterns are failing because they're not including this.

I was able to resolve this by modifying the feature pipelines to match by inserting (?: )? before the final $ in the failing grok patterns.

Our logs are missing the timezone so the HEADER pattern in logs-citrix-adc.log-1.6.0-native requires change to make it optional, as such: (?:%{WORD:event.timezone})?

I have not tested beyond these samples but looking at Citrix's Docs,this is likely the case. In any case where the log ends in variable.

I opened an issue on citrix's citrix/devdocs-issue-collector#36 a while back, after noticing there being no spaces where they should be. Now it seems that the case may be that a variable, such as %d would not be 1024 but 1024 , unsigned integers would replace %u with 1024, causing the double space when a value is empty, and the trailing one for those variables.

Sample event from our deployment

<134> 07/06/2024:19:30:07 fooserve 0-PPE-0 : default TCP CONN_TERMINATE 98222111 0 : Source 111.222.2.11:37080 - Destination 172.10.20.30:636 - Start Time 07/06/2024:19:30:07 - End Time 07/06/2024:19:30:07 - Total_bytes_send 261 - Total_bytes_recv 4944 \n"

Grok pattern in logs-citrix_adc.log-1.6.0-tcp_and_acl_feature

^Source %{IP:citrix_adc.log.source.ip}:%{INT:citrix_adc.log.source.port} - Destination %{IP:citrix_adc.log.destination.ip}:%{INT:citrix_adc.log.destination.port} - Start Time %{DATA:citrix_adc.log.start_time}(?: %{DATA:citrix_adc.log.start_time_timezone})? - End Time %{DATA:citrix_adc.log.end_time}(?: %{DATA:citrix_adc.log.end_time_timezone})? - Total_bytes_send %{INT:citrix_adc.log.total_bytes_send:long} - Total_bytes_recv %{INT:citrix_adc.log.total_bytes_received:long}$

Sample event from issue

The trailing space is an assumption here since the event appears to process fine without it.

<135> 06/14/2024:16:01:39 GMT host1 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_SUCCESS 220659228 0 : SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" - Session New - HandshakeTime 120 ms \n

Grok pattern in logs-citrix_adc.log-1.6.0-ssllog_feature

^SPCBId %{NUMBER:citrix_adc.log.spcb_id:int} - ClientIP %{IP:citrix_adc.log.client_ip} - ClientPort %{NUMBER:citrix_adc.log.client_port} - VserverServiceIP %{IP:citrix_adc.log.vserver.ip} - VserverServicePort %{NUMBER:citrix_adc.log.vserver.port} - ClientVersion %{DATA:citrix_adc.log.client_version} - CipherSuite \"%{GREEDYDATA:citrix_adc.log.cipher_suite}\"( - )?Session %{WORD:citrix_adc.log.session}(%{SPACE}- HandshakeTime %{INT:citrix_adc.log.handshake_time} ms)(?: - Reason \"%{GREEDYDATA:citrix_adc.log.reason}\")?$

Verifying with call to POST /_ingest/_simulate

Request

{
  "docs":[
  {
    "_index": ".ds-logs-citrix_adc.log-default-2024.06.28-000001",
    "_id": "123",
    "_source": {
      "log": {
        "source": {
          "address": "1.2.3.4:53577"
        }
      },
      "event": {
        "original": "<134> 07/06/2024:19:30:07  fooserve 0-PPE-0 : default TCP CONN_TERMINATE 98222111 0 :  Source 111.222.2.11:37080 - Destination 172.10.20.30:636 - Start Time 07/06/2024:19:30:07  - End Time 07/06/2024:19:30:07  - Total_bytes_send 261 - Total_bytes_recv 4944 \n"
      }
    }
  },
  {
    "_index": ".ds-logs-citrix_adc.log-default-2024.06.28-000001",
    "_id": "124",
    "_source": {
      "log": {
        "source": {
          "address": "1.2.3.4:53577"
        }
      },
      "event": {
        "original": "<135> 06/14/2024:16:01:39 GMT host1 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_SUCCESS 220659228 0 :  SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite \"TLS1.2-ECDHE-RSA-AES256-GCM-SHA384\" - Session New  - HandshakeTime 120 ms \n"
      }
    }
  }
]
 }

Result before making change

{
  "docs": [
    {
      "doc": {
        "_index": ".ds-logs-citrix_adc.log-default-2024.06.28-000001",
        "_version": "-3",
        "_id": "123",
        "_source": {
          "observer": {
            "product": "Netscaler",
            "type": "firewall",
            "vendor": "Citrix"
          },
          "ecs": {
            "version": "8.12.0"
          },
          "log": {
            "source": {
              "address": "1.2.3.4:53577"
            }
          },
          "event": {
            "kind": "pipeline_error",
            "timezone": "UTC"
          },
          "error": {
            "message": [
              "Processor grok with tag grok_detail in pipeline logs-citrix_adc.log-1.6.0-native failed with message: Provided Grok expressions do not match field value: [<134> 07/06/2024:19:30:07  fooserve 0-PPE-0 : default TCP CONN_TERMINATE 98222111 0 :  Source 111.222.2.11:37080 - Destination 172.10.20.30:636 - Start Time 07/06/2024:19:30:07  - End Time 07/06/2024:19:30:07  - Total_bytes_send 261 - Total_bytes_recv 4944 ]"
            ]
          },
          "citrix": {
            "cef_format": false,
            "detail": "<134> 07/06/2024:19:30:07  fooserve 0-PPE-0 : default TCP CONN_TERMINATE 98222111 0 :  Source 111.222.2.11:37080 - Destination 172.10.20.30:636 - Start Time 07/06/2024:19:30:07  - End Time 07/06/2024:19:30:07  - Total_bytes_send 261 - Total_bytes_recv 4944 "
          }
        },
        "_ingest": {
          "timestamp": "2024-07-07T00:06:34.02536742Z"
        }
      }
    },
    {
      "doc": {
        "_index": ".ds-logs-citrix_adc.log-default-2024.06.28-000001",
        "_version": "-3",
        "_id": "124",
        "_source": {
          "observer": {
            "product": "Netscaler",
            "type": "firewall",
            "vendor": "Citrix"
          },
          "@timestamp": "2024-06-14T16:01:39.000Z",
          "ecs": {
            "version": "8.12.0"
          },
          "log": {
            "source": {
              "address": "1.2.3.4:53577"
            }
          },
          "event": {
            "severity": 0,
            "id": "220659228",
            "category": [
              "network"
            ],
            "type": [
              "info"
            ],
            "timezone": "GMT"
          },
          "citrix": {
            "device_event_class_id": "SSLLOG",
            "host": "host1",
            "name": "SSL_HANDSHAKE_SUCCESS",
            "default_class": true,
            "detail": "<135> 06/14/2024:16:01:39 GMT host1 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_SUCCESS 220659228 0 :  SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite \"TLS1.2-ECDHE-RSA-AES256-GCM-SHA384\" - Session New  - HandshakeTime 120 ms ",
            "cef_format": false,
            "extended": {
              "message": "SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite \"TLS1.2-ECDHE-RSA-AES256-GCM-SHA384\" - Session New  - HandshakeTime 120 ms "
            }
          },
          "citrix_adc": {
            "log": {
              "message": "SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite \"TLS1.2-ECDHE-RSA-AES256-GCM-SHA384\" - Session New  - HandshakeTime 120 ms "
            }
          }
        },
        "_ingest": {
          "timestamp": "2024-07-07T00:06:34.025425921Z"
        }
      }
    }
  ]
}

Result after change

{
  "docs": [
    {
      "doc": {
        "_id": "123",
        "_index": ".ds-logs-citrix_adc.log-default-2024.06.28-000001",
        "_version": -3,
        "_source": {
          "observer": {
            "product": "Netscaler",
            "vendor": "Citrix",
            "type": "firewall"
          },
          "@timestamp": "2024-07-06T19:30:07.000Z",
          "ecs": {
            "version": "8.12.0"
          },
          "related": {
            "ip": [
              "111.222.2.11",
              "172.10.20.30"
            ]
          },
          "log": {
            "source": {
              "address": "1.2.3.4:53577"
            }
          },
          "destination": {
            "port": 636,
            "bytes": 4944,
            "ip": "172.10.20.30"
          },
          "source": {
            "geo": {
              "continent_name": "Asia",
              "country_iso_code": "CN",
              "country_name": "China",
              "location": {
                "lon": 113.722,
                "lat": 34.7732
              }
            },
            "as": {
              "number": 17962,
              "organization": {
                "name": "ShenZhen Topway Video Communication Co. Ltd"
              }
            },
            "port": 37080,
            "bytes": 261,
            "ip": "111.222.2.11"
          },
          "event": {
            "severity": 0,
            "agent_id_status": "missing",
            "ingested": "2024-07-07T01:16:43Z",
            "timezone": "UTC",
            "start": "2024-07-06T19:30:07.000Z",
            "end": "2024-07-06T19:30:07.000Z",
            "id": "98222111",
            "category": [
              "network"
            ],
            "type": [
              "end",
              "connection"
            ]
          },
          "citrix": {
            "device_event_class_id": "TCP",
            "host": "fooserve",
            "name": "CONN_TERMINATE",
            "default_class": true,
            "detail": "<134> 07/06/2024:19:30:07  fooserve 0-PPE-0 : default TCP CONN_TERMINATE 98222111 0 :  Source 111.222.2.11:37080 - Destination 172.10.20.30:636 - Start Time 07/06/2024:19:30:07  - End Time 07/06/2024:19:30:07  - Total_bytes_send 261 - Total_bytes_recv 4944 ",
            "cef_format": false,
            "extended": {
              "message": "Source 111.222.2.11:37080 - Destination 172.10.20.30:636 - Start Time 07/06/2024:19:30:07  - End Time 07/06/2024:19:30:07  - Total_bytes_send 261 - Total_bytes_recv 4944 "
            }
          },
          "citrix_adc": {
            "log": {
              "message": "Source 111.222.2.11:37080 - Destination 172.10.20.30:636 - Start Time 07/06/2024:19:30:07  - End Time 07/06/2024:19:30:07  - Total_bytes_send 261 - Total_bytes_recv 4944 "
            }
          }
        },
        "executed_pipelines": [
          "logs-citrix_adc.log-1.6.0",
          ".fleet_final_pipeline-1"
        ]
      }
    },
    {
      "doc": {
        "_id": "124",
        "_index": ".ds-logs-citrix_adc.log-default-2024.06.28-000001",
        "_version": -3,
        "_source": {
          "server": {
            "port": 443,
            "ip": "192.168.0.1"
          },
          "observer": {
            "product": "Netscaler",
            "vendor": "Citrix",
            "type": "firewall"
          },
          "@timestamp": "2024-06-14T16:01:39.000Z",
          "ecs": {
            "version": "8.12.0"
          },
          "related": {
            "ip": [
              "192.168.0.1",
              "1.1.1.1"
            ]
          },
          "log": {
            "source": {
              "address": "1.2.3.4:53577"
            }
          },
          "client": {
            "as": {
              "number": 13335,
              "organization": {
                "name": "CLOUDFLARENET"
              }
            },
            "port": 38797,
            "ip": "1.1.1.1"
          },
          "tls": {
            "cipher": "TLS1.2-ECDHE-RSA-AES256-GCM-SHA384",
            "version": "1.2",
            "version_protocol": "TLS"
          },
          "event": {
            "severity": 0,
            "agent_id_status": "missing",
            "ingested": "2024-07-07T01:16:43Z",
            "timezone": "GMT",
            "id": "220659228",
            "category": [
              "network"
            ],
            "type": [
              "info"
            ]
          },
          "citrix": {
            "device_event_class_id": "SSLLOG",
            "host": "host1",
            "name": "SSL_HANDSHAKE_SUCCESS",
            "default_class": true,
            "detail": """<135> 06/14/2024:16:01:39 GMT host1 0-PPE-0 : default SSLLOG SSL_HANDSHAKE_SUCCESS 220659228 0 :  SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" - Session New  - HandshakeTime 120 ms """,
            "cef_format": false,
            "extended": {
              "message": """SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" - Session New  - HandshakeTime 120 ms """
            }
          },
          "citrix_adc": {
            "log": {
              "spcb_id": "27719510",
              "session": "New",
              "handshake_time": "120",
              "message": """SPCBId 27719510 - ClientIP 1.1.1.1 - ClientPort 38797 - VserverServiceIP 192.168.0.1 - VserverServicePort 443 - ClientVersion TLSv1.2 - CipherSuite "TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" - Session New  - HandshakeTime 120 ms """,
              "client_version": "TLSv1.2"
            }
          }
        },
        "executed_pipelines": [
          "logs-citrix_adc.log-1.6.0",
          ".fleet_final_pipeline-1"
        ]
      }
    }
  ]
}