doc/userguide: more eve http upgrade notes

[jasonish]

Add more information with a examples of how the changes to EVE HTTP logging may affect users.

[jasonish]

To me this reinforces why we need to do this and I don't see a sane way to maintain compatibility. Lets take an arbitrary header name "Foo" that could exist in the request and response.

If we do:

"http": {
  "foo": "value",
}

was the server or response?

Further, what should we do about a header that appears multiple times. We then end up with:

"http": {
  "foo": ["value"],
}

(Elastic does fine with those arrays)

This may lead to one wanting:

"http": {
  "request_headers": {
    "foo": ["one", "two", "three"]
  }
}

but this conflicts with our existing request_headers and response_headers that are already arrays. We also can't make this objects for dump-all-headers as Elastic doesn't like arbitrary field growth, or certain chars in the keys.

As I mentioned over in https://redmine.openinfosecfoundation.org/issues/6173, we could make sure the "normalized" header names allowed in custom all have distinct names that don't overwrite any existing field name.. We should also prefix them with request_ and response_.

So for the custom: [Server] example I give, this would mean logging like:

"http": {
  "request_server": "nginx",
  ...
}

(could also be "request_server": ["nginx"] to handle multiple occurrences of a header.

This does feel like a bit of a hack, and will still break many reports etc. that will need to be updated for the new field names. But perhaps a little easier to adapt to than the array approach for 7.

[victorjulien]

I'm pretty sure libhtp won't allow key duplication in a single direction:

User-Agent: abc
User-Agent: def

will end up as

User-Agent: abc, def

Where abc, def is a single string.

[catenacyber]

I think this is rather Suricata doing a join(",") on libhtp's headers table

[jasonish]

So duplicate headers would be handled differently between http 1 and 2?

[catenacyber]

Good question, I think http2 has the same join(",") normalization at some point

But that needs to be tested ;-)

[catenacyber]

So looks like detection works on normalized with join(",") but logging logs only the last value 😨

[catenacyber]

Is this about @regit 's ticket ?

[jasonish]

Is this about @regit 's ticket ?

Yeah, but add more info just to make possible breakage more clear.

[jasonish]

Replaced by #9176.