[azure_network_watcher_vnet] Initial release of the azure network watcher vnet

[janvi-elastic]

Proposed commit message

Create New integration package azure_network_watcher_vnet.

• Added data stream.
• Added data collection logic for log data stream.
• Added the ingest pipeline for log data stream.
• Mapped fields according to the ECS schema and added Fields metadata in the appropriate yml files.
• Added dashboards and visualizations.
• Added test for pipeline for log data stream.
• Added system test cases for log data stream.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/azure_network_watcher_vnet directory.
• Run the following command to run tests.

elastic-package test

--- Test results for package: azure_network_watcher_vnet - START ---
╭────────────────────────────┬─────────────┬───────────┬───────────────┬────────┬──────────────╮
│ PACKAGE                    │ DATA STREAM │ TEST TYPE │ TEST NAME     │ RESULT │ TIME ELAPSED │
├────────────────────────────┼─────────────┼───────────┼───────────────┼────────┼──────────────┤
│ azure_network_watcher_vnet │ log         │ pipeline  │ test-vnet.log │ PASS   │    4.04771ms │
╰────────────────────────────┴─────────────┴───────────┴───────────────┴────────┴──────────────╯
--- Test results for package: azure_network_watcher_vnet - END   ---
Done
--- Test results for package: azure_network_watcher_vnet - START ---
╭────────────────────────────┬─────────────┬───────────┬──────────────────────────┬────────┬──────────────╮
│ PACKAGE                    │ DATA STREAM │ TEST TYPE │ TEST NAME                │ RESULT │ TIME ELAPSED │
├────────────────────────────┼─────────────┼───────────┼──────────────────────────┼────────┼──────────────┤
│ azure_network_watcher_vnet │ log         │ static    │ Verify sample_event.json │ PASS   │ 102.135753ms │
╰────────────────────────────┴─────────────┴───────────┴──────────────────────────┴────────┴──────────────╯
--- Test results for package: azure_network_watcher_vnet - END   ---
Done
--- Test results for package: azure_network_watcher_vnet - START ---
╭────────────────────────────┬─────────────┬───────────┬───────────┬────────┬───────────────╮
│ PACKAGE                    │ DATA STREAM │ TEST TYPE │ TEST NAME │ RESULT │  TIME ELAPSED │
├────────────────────────────┼─────────────┼───────────┼───────────┼────────┼───────────────┤
│ azure_network_watcher_vnet │ log         │ system    │ default   │ PASS   │ 37.890174557s │
╰────────────────────────────┴─────────────┴───────────┴───────────┴────────┴───────────────╯
--- Test results for package: azure_network_watcher_vnet - END   ---
Done
--- Test results for package: azure_network_watcher_vnet - START ---
╭────────────────────────────┬─────────────┬───────────┬─────────────────────────────────────────────────────────────────────────────────────┬────────┬──────────────╮
│ PACKAGE                    │ DATA STREAM │ TEST TYPE │ TEST NAME                                                                           │ RESULT │ TIME ELAPSED │
├────────────────────────────┼─────────────┼───────────┼─────────────────────────────────────────────────────────────────────────────────────┼────────┼──────────────┤
│ azure_network_watcher_vnet │             │ asset     │ dashboard azure_network_watcher_vnet-5ecce30b-1eba-4bd8-b440-cd266263449a is loaded │ PASS   │      1.093µs │
│ azure_network_watcher_vnet │             │ asset     │ search azure_network_watcher_vnet-0e5d4cba-9f41-4c45-b858-54d3e83df6e5 is loaded    │ PASS   │        118ns │
│ azure_network_watcher_vnet │             │ asset     │ search azure_network_watcher_vnet-51e7dbb1-2c87-429a-8ffa-f1c78ceb2eab is loaded    │ PASS   │        116ns │
│ azure_network_watcher_vnet │             │ asset     │ search azure_network_watcher_vnet-b52c9df2-cb7f-4ebe-af8b-569c213423cf is loaded    │ PASS   │        125ns │
│ azure_network_watcher_vnet │ log         │ asset     │ index_template logs-azure_network_watcher_vnet.log is loaded                        │ PASS   │        271ns │
│ azure_network_watcher_vnet │ log         │ asset     │ ingest_pipeline logs-azure_network_watcher_vnet.log-0.1.0 is loaded                 │ PASS   │        164ns │
╰────────────────────────────┴─────────────┴───────────┴─────────────────────────────────────────────────────────────────────────────────────┴────────┴──────────────╯
--- Test results for package: azure_network_watcher_vnet - END   ---
Done

Screenshot

[elasticmachine]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[kcreddy]

The PR description is following old format/template.
Please update PR description with repository PR template.

[kcreddy]

There are also other fields that can be derived from the resourceId path. Please refer to this pipeline: https://github.com/elastic/integrations/blob/main/packages/azure/data_stream/activitylogs/elasticsearch/ingest_pipeline/azure-shared-pipeline.yml and its corresponding fields https://github.com/elastic/integrations/blob/main/packages/azure/data_stream/activitylogs/fields/package-fields.yml.

[kcreddy]

@brijesh-elastic @janvi-elastic What I mean by this is you can apply similar grok processor on flow_log.resource_id field and extract azure* fields such as subscription_id, resource.group, resource.provider, etc,.

[kcreddy]

Since each of the below foreach processors are doing the same thing, it might be efficient to create one script processor to avoid multiple iterations of the same object/list.

[janvi-elastic]

We can add this processor in above script but for foreach processor which we have used to convert to long or IP we have to write foreach processor.

[kcreddy]

It maybe okay for now as its beta integration, but in the next enhancements before GA, this requires refactor. Essentially within your existing script after receiving flowTupleParts, you can check if it can be converted to long, only then append into respective map, if not append error.message.

integrations/packages/azure_network_watcher_vnet/data_stream/log/elasticsearch/ingest_pipeline/default.yml

Lines 163 to 164 in 64ed7ab

	for (String flowTuple: flowTuplesList) {
	String[] flowTupleParts = flowTuple.splitOnToken(',');

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: b9ef763

History

• 💚 Build #11021 succeeded 64ed7ab
• 💚 Build #10691 succeeded 5853dd7
• 💔 Build #10689 failed 36c563a

[elastic-sonarqube]

Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
92.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube

[kcreddy]

LGTM 👍🏼
Since this is similar to NSG PR, lets wait for Jamie's comments to be clarified: #9652 (comment) #9652 (comment)