Windows paths in List Format result in: "found unknown escape character. Exiting." or "found unexpected ':'. Exiting."

[PhaedrusTheGreek]

The certificate_authorities line in this configuration is invalid:

output:
  logstash:
    hosts: ["localhost:5045"]
    tls:
      certificate_authorities: [C:\Program Files\Filebeat\certs\filebeat.crt]
      certificate: C:\Program Files\Filebeat\certs\filebeat.crt
      certificate_key: C:\Program Files\Filebeat\certs\filebeat.key

Causes:

... found unexpected ':'. Exiting.

As is this:

      certificate_authorities: ["C:\Program Files\Filebeat\certs\filebeat.crt"]

Causes:

... found unknown escape character. Exiting.

The only way to get it to load is by escaping the backslashes, which seems inconsistent being that the only difference is that certificate_authorities requires a list format.

certificate_authorities: ["C:\\Program Files\\Filebeat\\certs\\filebeat.crt"]

Related: http://stackoverflow.com/questions/34513913/unable-to-start-filebeat-due-to-yaml-config-issue

[ruflin]

@urso Is this actually a bug or just a YAML problem?

[andrewkroh]

I think you can avoid this problem by using a forward slash in your Windows paths (Golang does the right thing and converts them IIRC) or instead of using double quotes, use single quotes.

[PhaedrusTheGreek]

@andrewkroh - this is confirmed that single quotes with forward slashes in arrays is accepted:

output:
  logstash:
    hosts: ["somehost:5045"]
    tls:
      certificate_authorities: ['C:/Program Files/Filebeat/certs/filebeat.crt']
      certificate: C:\Program Files\Filebeat\certs\filebeat.crt
      certificate_key: C:\Program Files\Filebeat\certs\filebeat.key

[andrewkroh]

I propose we resolve this issue by adding a special note to Windows users in our docs describing the options for specifying paths in YAML files. I was thinking to put it into the "Configuring *beat" section of the Getting Started guide for each beat, like this page for Filebeat. Or maybe an FAQ topic?

It would probably easiest to recommend using single quotes around any Windows path. That addresses any issue caused by the colon in absolute paths and avoids the need to escape any back-slashes.

// cc: @dedemorton

[dedemorton]

@andrewkroh It wouldn't hurt to mention this in a few places: in the config section of each GS and where we mention specifying a path. An FAQ topic is also a good idea, especially if "found unknown escape character" generally means that you've specified a Windows path that uses backslashes. I'll assign this item to myself and play it by ear (don't want to repeat this a bazillion times, but wan to make sure that users don't overlook it).

[urso]

I'm advocating use of single-quotes for paths and regular expressions for quite some time now.
YAML has 5 different kind of strings, each with slightly different semantics (yay). Using single quotes ensures the YAML parser is not interpreting the string-contents in any way. As long as you don't need a ' symbol in your string, always use single-quotes for things like paths and regular expressions. Otherwise you have to use double-quotes with additional escaping.

Windows path with C:/ ... add another quirk/ambiguity... as raw unquoted string followed by : must be parsed as dictionary in YAML (according to spec). The spec/parser is not guessing between windows-file-path and valid YAML dictionary. That is, windows paths starting with a drive letter must always be quoted.

All this behaviour is by (YAML) design/specification. For as long as we're using YAML, the best solution is to use single-quotes consistently.

[dedemorton]

I've added #3882 to address this problem

The following dev PR should also help users: #3781