builder: use structured logging for maven output

[lburgazzoli]

http://development.wombatsecurity.com/development/2018/04/06/logging-maven-to-json/

[lburgazzoli]

//cc @astefanutti

[astefanutti]

It complements #580 for integration logging.

[orpiske]

I am looking at this.

[orpiske]

So, I got that working. The original link does not work, but the page is archived on wayback machine.

There are a few caveats, though:

1. We lose the old non-parseable Maven output, since we have to remove the maven slf4j adapter
2. The container is slightly bigger
3. The build process is slightly longer and more prune to fail due to downloading extra dependencies for maven itself
4. Although it outputs in json format, it's not the same json format as the one in the operator logs. We may need to tune it - or allow the user to tune it - so it can be parseable.

[astefanutti]

Great! I think these are acceptable, compared to having the operator log fully JSONified.

Just to be sure, what do you mean exactly by 4?

The first value is to have full JSON for the operator log. Then, the second requirement would be to enable correlation for the Build logs, that are interleaved into the operator reconcile loop. To meed that requirement, it'd be useful to add a build correlation key. Or to find another mechanism, so that SOPs or users can quickly pick into the build and reconciliation for a particular resource.

[orpiske]

Thanks!

Great! I think these are acceptable, compared to having the operator log fully JSONified.

Just to be sure, what do you mean exactly by 4?

Basically, what I meant is that the JSON for the build log and the JSON for the operator logs are different.

{"level":"info","ts":1620106986.5145833,"logger":"camel-k.controller.kamelet","msg":"Invoking action monitor","request-namespace":"camel-colorization","request-name":"cassandra-sink","api-version":"camel.apache.
org/v1alpha1","kind":"Kamelet","ns":"camel-colorization","name":"cassandra-sink"}

The maven logs are slightly different and we'll have to adjust that to make them look the same (either by modifying the logback configuration or consuming them).

The first value is to have full JSON for the operator log. Then, the second requirement would be to enable correlation for the Build logs, that are interleaved into the operator reconcile loop. To meed that requirement, it'd be useful to add a build correlation key. Or to find another mechanism, so that SOPs or users can quickly pick into the build and reconciliation for a particular resource.

I see. It makes sense. If I understand it correctly, this is just the first part: just printing the maven logs in JSON format.

To make it truly useful, we'll have to consume and process that maven JSON format so we can normalize it and correlate it. So, IMHO, the next step would be to work on that.

Do you think it makes sense to split that into 2 tasks (and, consequently, in 2 contributions)?

[lburgazzoli]

2. The container is slightly bigger

3. The build process is slightly longer and more prune to fail due to downloading extra dependencies for maven itself

Cane we eventually make a pre-build container image with all the required bits already there ?

[astefanutti]

@orpiske

Do you think it makes sense to split that into 2 tasks (and, consequently, in 2 contributions)?

Whatever works best for you. Having the Maven logs JSONified is already an improvement, so it can be merged and have it followed by other contributions.

[astefanutti]

@lburgazzoli

Can we eventually make a pre-build container image with all the required bits already there?

My understanding is that the image here is the operator one. So it'll be updated to have all the bits.

[orpiske]

Whatever works best for you. Having the Maven logs JSONified is already an improvement, so it can be merged and have it followed by other contributions.

Deal, thanks. I'll clean it up and send the current implement today for review.

@lburgazzoli

Can we eventually make a pre-build container image with all the required bits already there?

My understanding is that the image here is the operator one. So it'll be updated to have all the bits.

That's correct.

[astefanutti]

Fixed with #2262.