Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Network Packet Capture] Convert Dashboards to lens, add pipeline errors and format_versions #6117

Merged
merged 28 commits into from
May 15, 2023
Merged

[Network Packet Capture] Convert Dashboards to lens, add pipeline errors and format_versions #6117

merged 28 commits into from
May 15, 2023

Conversation

P1llus
Copy link
Member

@P1llus P1llus commented May 8, 2023 •
edited
Loading

What does this PR do?

This is part of a bigger cleanup that relates to https://github.com/elastic/security-team/issues/5910.

The tasks, when applicable was:

  • Update format_version to 2.6.0+
  • Update on_failure to include proper event.kind.
  • Update on_failure error message to be more descriptive
  • Add processor tags for on_failure message
  • If system/pipeline tests are missing, add them when possible. (Removed one broken)
  • If dashboards exist, replace with lens when needed, update dashboards if applicable.

Dashboard changes:

  1. All table of contents.
  2. Convert all to lens.
  3. Add/remove/rename certain visualizations to fit the layout and usefulness better
  4. Changed layout a bit.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Testing

Use commit 70d11d1 for testing
I have temporarily added interface options to each datastream to help with testing, it will be removed before merging. The steps to test this is:

  1. Copy the PCAP files not including the folder, from the _dev/deploy/docker/pcaps/* folder to ~/.elastic-package/tmp/service_logs/*
  2. Bring up the stack (version 8.6.2).
  3. Once everything is up and healthy, you can run the below snippet in Kibana Dev tools. It will update the agent policy to enable all protocols, and point them to their respective pcap files.
    One note, is that since we have to use a direct reference to the pcap files, if you want to ingest more test-data, you would have to manually update the agent policy, and change the path to another pcap, save, wait for the ingest to finish, then change it again etc.

EDIT: Updated step 1, to make it more specific.

Test policy:

POST kbn:/api/fleet/package_policies
{
  "policy_id": "elastic-agent-managed-ep",
  "package": {
    "name": "network_traffic",
    "version": "1.16.0"
  },
  "name": "network_traffic-1",
  "description": "",
  "namespace": "default",
  "inputs": {
    "network-packet": {
      "enabled": true,
      "vars": {
        "never_install": false
      },
      "streams": {
        "network_traffic.amqp": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/amqp_channel_error.pcap",
            "port": [
              5672
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.cassandra": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/cassandra_mixed_frame.pcap",
            "port": [
              9042
            ],
            "geoip_enrich": true,
            "ignored_ops": [],
            "tags": []
          }
        },
        "network_traffic.dhcpv4": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/dhcp.pcap",
            "port": [
              67,
              68
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.dns": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/dns_ns.pcap",
            "port": [
              53
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.flow": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/http_10_connection_close.pcap",
            "geoip_enrich": true,
            "period": "10s",
            "timeout": "30s",
            "tags": []
          }
        },
        "network_traffic.http": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/http_10_connection_close.pcap",
            "port": [
              80,
              8080,
              8000,
              5000,
              8002
            ],
            "hide_keywords": [],
            "send_headers": [],
            "redact_headers": [],
            "include_body_for": [],
            "include_request_body_for": [],
            "include_response_body_for": [],
            "tags": []
          }
        },
        "network_traffic.icmp": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/icmp4_ping_over_vlan.pcap",
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.memcached": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/memcache_bin_tcp_stats.pcap",
            "port": [
              11211
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.mongodb": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/mongo_3.0_session.pcap",
            "port": [
              27017
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.mysql": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/mysql_long.pcap",
            "port": [
              3306,
              3307
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.nfs": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/nfs_v4.pcap",
            "port": [
              2049
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.pgsql": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/pgsql_long_result.pcap",
            "port": [
              5432
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.redis": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/redis_session.pcap",
            "port": [
              "6380"
            ],
            "geoip_enrich": true,
            "tags": []
          }
        },
        "network_traffic.sip": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/sip.pcap",
            "port": [
              5060
            ],
            "geoip_enrich": true,
            "use_tcp": false,
            "tags": []
          }
        },
        "network_traffic.thrift": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/thrift_integration.pcap",
            "port": [
              9090
            ],
            "geoip_enrich": true,
            "idl_files": [],
            "tags": []
          }
        },
        "network_traffic.tls": {
          "enabled": true,
          "vars": {
            "interface": "/tmp/service_logs/tls-version-13.pcap",
            "port": [
              443,
              993,
              995,
              5223,
              8443,
              8883,
              9243
            ],
            "geoip_enrich": true,
            "fingerprints": [],
            "tags": []
          }
        }
      }
    }
  }
}

Related issues

Screenshots (Somewhat limited due to test data available, but working fine).

Overview
newOverview1
newOverview2

Mysql:
newMYSQL1
newMYSQL2

NFS:
newNFS1
newNFS2

PGSQL:
newpg1
newpg2

Thrift:
newThrift1
newThrift2

Cassandra:
newCassandra1

DHCP:
newDHCP1

DNS:
newDNS1
newDNS2

HTTP:
newHTTP1
newHTTP2

MongoDB:
newMongo1
newMongo2
newMongo3

@P1llus
Copy link
Member Author

P1llus commented May 8, 2023

Initial Draft added everything except dashboards, next commits will add them in, will keep it in draft for now

@P1llus P1llus changed the title Package network capture dashboardupdate2 [Network Packet Capture] Convert Dashboards to lens, add pipeline errors and format_versions May 8, 2023
@elasticmachine
Copy link

elasticmachine commented May 8, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-05-15T10:51:28.889+0000

  • Duration: 58 min 54 sec

Test stats 🧪

Test Results
Failed 0
Passed 153
Skipped 0
Total 153

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented May 8, 2023 •
edited
Loading

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (0/0) 💚
Files 100.0% (0/0) 💚
Classes 100.0% (0/0) 💚
Methods 75.0% (48/64) 👎 -21.154
Lines 100.0% (0/0) 💚
Conditionals 100.0% (0/0) 💚

@P1llus P1llus marked this pull request as ready for review May 10, 2023 12:03
@P1llus P1llus requested a review from a team as a code owner May 10, 2023 12:03
@elasticmachine
Copy link

Pinging @elastic/security-external-integrations (Team:Security-External Integrations)

@P1llus
Copy link
Member Author

P1llus commented May 10, 2023

This does not include data views yet, to fix field collisions with Cloud Securiry Posture, will look into that before merging.

P1llus
P1llus commented May 11, 2023
View reviewed changes
packages/network_traffic/data_stream/amqp/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/cassandra/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/dhcpv4/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/dns/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/flow/manifest.yml Outdated
@@ -6,6 +6,11 @@ streams:
description: Track Network Flows
template_path: flow.yml.hbs
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/pgsql/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/redis/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/sip/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/thrift/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

packages/network_traffic/data_stream/tls/manifest.yml Outdated
@@ -3,6 +3,11 @@ type: logs
streams:
- input: packet
vars:
- name: interface
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be removed before merge, its to help with people testing, see PR description.
While we would have wanted this to be included, apparently packetbeat do not support different interfaces for different protocols, but it does work with pcaps.

@P1llus
Copy link
Member Author

P1llus commented May 11, 2023

Added new and better screenshots, Tests are most likely failing due to the test specific commit I added in, it will go back to green once its removed (once people have had the oppertunity to test)

@P1llus
Copy link
Member Author

P1llus commented May 11, 2023

Seems like some of the filters even references dataviews, will have to re-add all filters to remove any reference to logs-*

@P1llus P1llus merged commit c9d9afc into elastic:main May 15, 2023
@elasticmachine
Copy link

Package network_traffic - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=network_traffic

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@efd6 efd6 efd6 approved these changes

Assignees
No one assigned
Labels
enhancement New feature or request Integration:network_traffic Network Packet Capture
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@P1llus @elasticmachine @efd6