[Merged by Bors] - feat(new sink): Initial `azure_monitor_logs` sink #2811

nazar554 · 2020-06-13T15:51:55Z

Solves #1808
Will add validations for configuration and unit tests

nazar554 · 2020-06-13T19:05:22Z

Right now I assume by default that events have timestamp field that is ISO8601. Is there a way to make sure that it's the correct field and it's not using Unix format?

There seems to be an issue when a single event is batched in a JSON array. Azure seems to ignore the timestamp field, maybe because I don't drop milliseconds (although it works fine when there are two or more events)

lukesteensen

This looks really good, thanks @nazar554!

Is there some local version of this service that we could add to our integration tests?

We'll also want to add docs, but that can happen in a followup PR.

lukesteensen · 2020-06-16T23:17:27Z

Right now I assume by default that events have timestamp field that is ISO8601. Is there a way to make sure that it's the correct field and it's not using Unix format?

By default, events get a timestamp field with an actual Timestamp value, not just a string. But that can often be overwritten with user data that can be a string, so it's hard to guarantee anything. Most sinks will read that field and check the type before using it, which it looks like you're doing.

There seems to be an issue when a single event is batched in a JSON array. Azure seems to ignore the timestamp field, maybe because I don't drop milliseconds (although it works fine when there are two or more events)

That is odd. I didn't see anything that should behave differently with an array of one event.

nazar554 · 2020-06-17T11:15:08Z

@lukesteensen

Is there some local version of this service that we could add to our integration tests?

I don't know one, you need to have a Azure Log Analytics service in order to obtain shared_key and customer_id. However, there is free 5 Gb per month plan, so if you have an Azure subscription then it's possible to test this for fee.

That is odd. I didn't see anything that should behave differently with an array of one event.

I think it's the behavior of the Azure Log Analytics. Official documentation suggests a YYYY-MM-DDThh:mm:ssZ date format, but if you actually view some logs on Azure Portal it seems they have milliseconds there. It's possible that docs were not updated.
By default impl Serialize does nanosecond precision and this makes Azure drop the timestamp entirely (it uses the ingestion time instead).
So I added one more commit where I format the date manually and use chrono::SecondsFormat::Millis.

We'll also want to add docs, but that can happen in a followup PR.

I think I can add docs as well, I'll try to use some existing sink as a reference

jamtur01 · 2020-08-21T20:22:22Z

@nazar554 Just thought I'd bump this and see if you have got a moment to re-visit. Thanks! 🥇

nazar554 · 2020-08-21T21:04:24Z

@jamtur01 Was a bit busy but I think I can get back to this sometime next week

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

…ay, adjust headers Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

…amp precision Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

nazar554 · 2020-08-23T22:15:43Z

@jamtur01 I am not sure how to build & test the docs, should I clone and build timberio/vector-website ?

jamtur01 · 2020-08-23T22:25:25Z

@nazar554 I am not 100% sure how to test off a branch - @binarylogic?

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

nazar554 · 2020-08-28T17:18:35Z

Anything else I need to do here? Should I also update vector.spec.toml?

lukesteensen · 2020-08-31T19:49:38Z

@nazar554 That file gets updated automatically from the config definitions in .meta/, so you should be good there.

Once @binarylogic is able to help out with testing the docs I'll make another pass and see if there are any remaining blockers. Thanks again for opening this! Excited to get it merged.

binarylogic · 2020-08-31T19:51:45Z

@nazar554 the docs look good. As long as make check-meta works (which is tested in the CI check workflow), then we're good. No need to do anything with the generated files.

lukesteensen · 2020-10-02T22:27:11Z

@binarylogic it looks like something is up with check-meta here. Is there something that needs to be updated in the docs?

jamtur01

One minor issue.

jamtur01 · 2020-10-02T22:28:00Z

.meta/sinks/azure_monitor_logs.toml.erb

+
+<%= render(
+  "_partials/fields/_tls_connector_options.toml",
+  namespace: "sinks.azure_monitor_logs.options",


This needs:

enabled_default: false,

With the appropriate true or false value. That'll fix the failing Lint suite test.

lukesteensen

This is looking good to me! If we can get CI green then I think we're ready to merge.

Thanks for your patience with this @nazar554, and apologies for it lingering so long. We really appreciate the contribution!

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

jamtur01

Looks like some test failures in the CI run.

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

jamtur01

LGTM.

jamtur01 · 2020-10-03T23:33:40Z

@nazar554 I just merged master into the branch because there are some CI changes and I want to make sure the merge goes smooth. Will merge post that.

jamtur01 · 2020-10-04T00:09:05Z

bors merge

Solves #1808 Will add validations for configuration and unit tests Co-authored-by: Luke Steensen <luke.steensen@gmail.com> Co-authored-by: James Turnbull <james@lovedthanlost.net>

bors · 2020-10-04T00:09:40Z

Pull request successfully merged into master.

Build succeeded:

ci

Solves vectordotdev#1808 Will add validations for configuration and unit tests Co-authored-by: Luke Steensen <luke.steensen@gmail.com> Co-authored-by: James Turnbull <james@lovedthanlost.net> Signed-off-by: Brian Menges <brian.menges@anaplan.com>

binarylogic requested a review from lukesteensen June 13, 2020 15:59

nazar554 marked this pull request as ready for review June 13, 2020 17:35

lukesteensen reviewed Jun 16, 2020

View reviewed changes

jamtur01 added the needs: docs Needs documentation updates label Aug 21, 2020

jamtur01 assigned nazar554 Aug 21, 2020

nazar554 added 5 commits August 23, 2020 21:37

Add azure_monitor_logs sink

f15af1a

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

azure_monitor_logs: Add unit tests, fix encoding to remove double arr…

0c19fc3

…ay, adjust headers Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

azure_monitor_logs: Sort features alphabetically, run rustfmt

d8f3c7f

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

azure_monitor_logs: Remove time_generated_field option, adjust timest…

8f592db

…amp precision Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

azure_monitor_logs: Add documentation

217e580

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

nazar554 requested a review from lukesteensen August 23, 2020 19:49

nazar554 added 2 commits August 24, 2020 00:53

azure_monitor_logs: Fix build & lint errors

9244161

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

azure_monitor_logs: Fix building unit tests

b8e6ed8

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

azure_monitor_logs: Fix clippy lint in unit tests

528bcd9

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

binarylogic self-assigned this Aug 23, 2020

jamtur01 added this to the 2020-08-31 - Digitization Laser milestone Sep 11, 2020

binarylogic modified the milestones: 2020-08-31 - Digitization Laser, 2020-09-14 - The Grid Sep 14, 2020

jamtur01 removed this from the 2020-09-14 - The Grid milestone Sep 28, 2020

jamtur01 self-assigned this Sep 29, 2020

jamtur01 unassigned binarylogic Sep 29, 2020

jamtur01 removed the needs: docs Needs documentation updates label Sep 29, 2020

Merge branch 'master' into feature/azure-monitor-logs

ec92c1d

jamtur01 suggested changes Oct 2, 2020

View reviewed changes

lukesteensen approved these changes Oct 2, 2020

View reviewed changes

Add enabled_default for azure_monitor_logs tls_connector_options

ace34e5

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

nazar554 requested a review from jamtur01 October 3, 2020 16:44

jamtur01 suggested changes Oct 3, 2020

View reviewed changes

nazar554 added 2 commits October 4, 2020 00:43

azure_monitor_logs: Fix event::log_schema()

bf68b70

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

azure_monitor_logs: fix build failure

f5512ad

Signed-off-by: Nazar Mishturak <nazarmx@gmail.com>

jamtur01 approved these changes Oct 3, 2020

View reviewed changes

Merge branch 'master' into feature/azure-monitor-logs

3b8715f

bors bot changed the title ~~feat(new sink): Initial azure_monitor_logs sink~~ [Merged by Bors] - feat(new sink): Initial azure_monitor_logs sink Oct 4, 2020

bors bot closed this Oct 4, 2020

nazar554 deleted the feature/azure-monitor-logs branch October 4, 2020 13:27

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Merged by Bors] - feat(new sink): Initial `azure_monitor_logs` sink #2811

[Merged by Bors] - feat(new sink): Initial `azure_monitor_logs` sink #2811

nazar554 commented Jun 13, 2020

nazar554 commented Jun 13, 2020

lukesteensen left a comment

lukesteensen commented Jun 16, 2020

nazar554 commented Jun 17, 2020

jamtur01 commented Aug 21, 2020

nazar554 commented Aug 21, 2020

nazar554 commented Aug 23, 2020

jamtur01 commented Aug 23, 2020

nazar554 commented Aug 28, 2020

lukesteensen commented Aug 31, 2020

binarylogic commented Aug 31, 2020

lukesteensen commented Oct 2, 2020

jamtur01 left a comment

jamtur01 Oct 2, 2020 •

edited

Loading

lukesteensen left a comment

jamtur01 left a comment

jamtur01 left a comment

jamtur01 commented Oct 3, 2020

jamtur01 commented Oct 4, 2020

bors bot commented Oct 4, 2020

[Merged by Bors] - feat(new sink): Initial azure_monitor_logs sink #2811

[Merged by Bors] - feat(new sink): Initial azure_monitor_logs sink #2811

Conversation

nazar554 commented Jun 13, 2020

nazar554 commented Jun 13, 2020

lukesteensen left a comment

Choose a reason for hiding this comment

lukesteensen commented Jun 16, 2020

nazar554 commented Jun 17, 2020

jamtur01 commented Aug 21, 2020

nazar554 commented Aug 21, 2020

nazar554 commented Aug 23, 2020

jamtur01 commented Aug 23, 2020

nazar554 commented Aug 28, 2020

lukesteensen commented Aug 31, 2020

binarylogic commented Aug 31, 2020

lukesteensen commented Oct 2, 2020

jamtur01 left a comment

Choose a reason for hiding this comment

jamtur01 Oct 2, 2020 • edited Loading

Choose a reason for hiding this comment

lukesteensen left a comment

Choose a reason for hiding this comment

jamtur01 left a comment

Choose a reason for hiding this comment

jamtur01 left a comment

Choose a reason for hiding this comment

jamtur01 commented Oct 3, 2020

jamtur01 commented Oct 4, 2020

bors bot commented Oct 4, 2020

[Merged by Bors] - feat(new sink): Initial `azure_monitor_logs` sink #2811

[Merged by Bors] - feat(new sink): Initial `azure_monitor_logs` sink #2811

jamtur01 Oct 2, 2020 •

edited

Loading