Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add global level data_stream.daataset dashboard filter #9699

Closed
wants to merge 2 commits into from
Closed

Add global level data_stream.daataset dashboard filter #9699

wants to merge 2 commits into from

Conversation

milan-elastic
Copy link
Contributor

@milan-elastic milan-elastic commented Apr 24, 2024 •
edited

Proposed commit message

Add a global filter to the dashboard and remove the subfilters on the dataset and a global filter on the dataset is introduced. Filters on data_stream.dataset are applied with a phrase type filter on the dataset that is being used in that particular dashboard.

Expect the above mentioned changes done in following selected packages to ease the review

  • ActiveMQ
  • Apache Spark
  • Apache Tomcat
  • Hadoop

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Related issues

Screenshot

ActiveMQ
Broker Before Broker After
activemq broker - before activemq broker - after
Topic Before Topic After
activemq topic - before activemq topic - after
Logs Before Logs After
activemq logs - before activemq logs - after
Queue Before Queue After
ActiveMQ Queue - before activemq queue - after
Audit Before Audit After
activemq audit - before activemq audit - after
Apache Spark
Overview Before Overview After
apache spark overview - before apache spark overview - after
Apache Tomcat
Session Before Session After
apache tomcat session - before apache tomcat session - after
Metrics Overview Before Metrics Overview After
apache tomcat metrics overview - before apache tomcat metrics overview - after
Before After
apache tomcat rquest - before apache tomcat request - after
Memory Before Memory After
apache tomcat memory - before apache tomcat memory - after
Cache Before Cache After
apache tomcat cache - before apache tomcat cache - after
Threadpool Before Threadpool After
apache tomcat thread pool - before apache tomcat thread pool - after
Connection pool Before Connection pool After
apache tomcat connection pool - before apache tomcat connection pool -after
Hadoop
Datanode Before Datanode After
before_hadoop_datanode after_hadoop_datanode
Node Manager Before Node Manager After
before_hadoop_node_manager after_hadoop_node_manager
Cluster Before Cluster After
before_hadoop_cluster after_hadoop_cluster
Application Before Application After
before_hadoop_application after_hadoop_application

@elasticmachine
Copy link

elasticmachine commented Apr 24, 2024 •
edited

🚀 Benchmarks report

Package bluecoat 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
director 37037.04 19607.84 -17429.2 (-47.06%) 💔

Package carbonblack_edr 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1303.78 845.31 -458.47 (-35.16%) 💔

Package f5 👍(0) 💚(0) 💔(2)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
bigipafm 34482.76 28571.43 -5911.33 (-17.14%) 💔
bigipapm 20408.16 11764.71 -8643.45 (-42.35%) 💔

Package fortinet_forticlient 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 17241.38 12658.23 -4583.15 (-26.58%) 💔

Package fortinet_fortiedr 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1070.66 508.65 -562.01 (-52.49%) 💔

Package keycloak 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 1184.83 946.07 -238.76 (-20.15%) 💔

Package mattermost 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
audit 1142.86 787.4 -355.46 (-31.1%) 💔

Package microsoft_dhcp 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 16393.44 10309.28 -6084.16 (-37.11%) 💔

Package modsecurity 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
auditlog 673.4 362.71 -310.69 (-46.14%) 💔

Package squid 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 38461.54 23255.81 -15205.73 (-39.53%) 💔

Package symantec_endpoint 👍(0) 💚(0) 💔(1)

Expand to view
Data stream Previous EPS New EPS Diff (%) Result
log 487.33 368.87 -118.46 (-24.31%) 💔

To see the full report comment with /test benchmark fullreport

@cla-checker-service CLA Checker Service
Copy link

cla-checker-service bot commented May 1, 2024 •
edited

💚 CLA has been signed

@milan-elastic
Squashed commit of the following:
8cf8f03 
commit e2a688fbb1c8712ba0cad243713146867ac2f986
Author: milan-elastic <milan.Parmar@elastic.co>
Date:   Wed May 1 15:43:52 2024 +0530

    Squashed commit of the following:

    commit a17de73
    Author: milan-elastic <“milan.parmar@elastic.co”>
    Date:   Wed May 1 15:29:41 2024 +0530

        Squashed commit of the following:

        commit fccdb1f
        Author: milan-elastic <milan.parmar@elastic.co>
        Date:   Wed May 1 14:58:41 2024 +0530

            add global filter on dashboard level for hadoop

        commit 686e49b
        Merge: 024d864 01201a7
        Author: “milan-elastic” <“milan.parmar@elastic.co”>
        Date:   Wed May 1 11:38:59 2024 +0530

            Merge branch 'main' of github.com:milan-elastic/integrations into mongodb-atlas-database-logs

        commit 01201a7
        Author: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
        Date:   Tue Apr 30 10:46:55 2024 -0400

            [Security Rules] Update security rules package to v8.13.5 (elastic#9762)

            * [Security Rules] Update security rules package to v8.13.5

            * Add changelog entry for 8.13.5

            ---------

            Co-authored-by: protectionsmachine <72879786+protectionsmachine@users.noreply.github.com>

        commit c9d1f1b
        Author: Eric Forte <119343520+eric-forte-elastic@users.noreply.github.com>
        Date:   Tue Apr 30 09:30:30 2024 -0400

            [Security Rules] Update security rules package to v8.13.5-beta.1 (elastic#9758)

            * [Security Rules] Update security rules package to v8.13.5-beta.1

            * Add changelog entry for 8.13.5-beta.1

            ---------

            Co-authored-by: protectionsmachine <72879786+protectionsmachine@users.noreply.github.com>

        commit a79f813
        Author: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>
        Date:   Tue Apr 30 11:32:37 2024 +0200

            [kubernetes] Remove deprecated fields, add missing status.last_terminated_reason metric (elastic#9736)

            * remove deprecated fields

            Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

            * Update changelog.yml

            * add missing metric: last_terminated_reason; update description of the status.reason field

            Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

            ---------

            Signed-off-by: Tetiana Kravchenko <tetiana.kravchenko@elastic.co>

        commit b1627a3
        Author: ShourieG <105607378+ShourieG@users.noreply.github.com>
        Date:   Tue Apr 30 13:03:29 2024 +0530

            [integrations][http_endpoint] - Converted HTTP Endpoint Integration to input type (elastic#9732)

            * converted http_endpoint to input package type

            * updated changelog

            * updated original event in sample event

        commit 3a9b508
        Author: Lalit Satapathy <69236064+lalit-satapathy@users.noreply.github.com>
        Date:   Tue Apr 30 11:49:09 2024 +0530

            Remove separate codeowners for system package kibana paths. (elastic#9731)

        commit c90e817
        Author: Krishna Chaitanya Reddy Burri <krishnachaitanyareddy.burri@elastic.co>
        Date:   Tue Apr 30 11:32:17 2024 +0530

            [Crowdstrike,Azure] Fix flaky tests with ECS fields (elastic#9738)

            * Fix flaky pipeline tests.

               * `azure.graphactivitylogs`: Add missing ECS field definitions.

               * `crowdstrike.falcon`: Update `geoip` processor to `destination` instead of `source`.

        commit ace8fb4
        Author: Aliabbas Attarwala <124054599+aliabbas-elastic@users.noreply.github.com>
        Date:   Mon Apr 29 16:37:23 2024 +0530

            [O11y][AWS] Rally benchmark `aws.cloudtrail` (elastic#9448)

        commit d4e4aa4
        Author: niraj-elastic <124254029+niraj-elastic@users.noreply.github.com>
        Date:   Mon Apr 29 14:45:46 2024 +0530

            [Apache] Update grok pattern for accepting user-identity (elastic#9632)

            * update grok pattern

            * update changelog

            * address review comments

            * address review comments

            Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

            * address review comments

            * address review comment

            ---------

            Co-authored-by: muthu-mps <101238137+muthu-mps@users.noreply.github.com>

        commit dce5699
        Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
        Date:   Mon Apr 29 10:33:19 2024 +0200

            Enable publishing packages from integrations-publish pipeline (elastic#9712)

            Enable publishing packages from integrations-publish pipeline,
            and remove corresponding step from the main pipeline.

        commit c7bc530
        Author: Chema Martínez <chema.martinez@elastic.co>
        Date:   Sat Apr 27 08:57:55 2024 +0200

            [zscaler_zia] Fix mapping of source.ip and source.nat.ip (elastic#9727)

            * Fix mapping of source.ip and source.nat.ip

            * Update changelog

            * updated web datastream pipeline tests

            ---------

            Co-authored-by: Shourie Ganguly <shourie.ganguly@elastic.co>

        commit 4750ea8
        Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
        Date:   Fri Apr 26 13:09:53 2024 +0200

            [nginx] Update nginx config to listen in ipv6 too (elastic#9720)

        commit 25b0988
        Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
        Date:   Fri Apr 26 10:45:03 2024 +0200

            [Buildkite] Update filter to use api source (elastic#9717)

        commit 45327cf
        Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
        Date:   Fri Apr 26 10:13:22 2024 +0200

            [Buildkite] Update filter condition to allow just from webhook source (elastic#9714)

        commit 024d864
        Author: milan-elastic <milan.parmar@elastic.co>
        Date:   Fri Apr 26 13:00:47 2024 +0530

            add dashboard level filter for apache tomcat

        commit 1cb5fad
        Author: Dan Kortschak <dan.kortschak@elastic.co>
        Date:   Fri Apr 26 16:23:35 2024 +0930

            entityanalytics_ad: new package for Active Directory user collection (elastic#9485)

        commit 37c598f
        Author: CarsonHrusovsky <95260807+CarsonHrusovsky@users.noreply.github.com>
        Date:   Thu Apr 25 18:13:26 2024 -0500

            [BBOT] New integration for Black Lantern Security scanner (elastic#9651)

        commit d13e474
        Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
        Date:   Thu Apr 25 11:55:39 2024 +0200

            [Buildkite] Skip install package command in serverless builds for some packages (elastic#9686)

        commit 0c2198b
        Author: Mario Rodriguez Molins <mario.rodriguez@elastic.co>
        Date:   Thu Apr 25 11:41:42 2024 +0200

            [Buildkite] Add retry suffix for logs (elastic#9703)

        commit d932e79
        Author: Simon Kötting <145989254+SimonKoetting@users.noreply.github.com>
        Date:   Thu Apr 25 07:35:45 2024 +0200

            [Exchange Server] GA of Integration, Add Dashbord Panel Titles & System Tests (elastic#9560)

            * Add Dashboard Titles

            * Add Dashboard Titles

            * Change Version to GA

            * adjust PR in Changelog

            * Add System Tests to all datstreams

            * fix imap system test config

            * remove Folder structure out of system tests sample logs

            * Fix mapping

            * Add convert for inode field

            * specify numeric_keyword_fields in system tests

        commit dba2901
        Author: Dan Kortschak <dan.kortschak@elastic.co>
        Date:   Thu Apr 25 10:21:30 2024 +0930

            rapid7_insightvm: canonicalize host.name to lower case and map subdomain to host.hostname (elastic#9665)

        commit 4284262
        Author: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>
        Date:   Wed Apr 24 20:34:13 2024 +0300

            fix(fim): add auto option for backend and make it the default one (elastic#9702)

        commit c563bb3
        Author: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>
        Date:   Wed Apr 24 19:40:04 2024 +0300

            [juniper_netscreen]: include log.file.device_id and log.file.inode in base-fields (elastic#9658)

            * fix(juniper_netscreen): include log.file.device_id and log.file.inode in base-fields.yml

            * fix(juniper_netscreen): update README.md

        commit f187d0d
        Author: Panos Koutsovasilis <panos.koutsovasilis@elastic.co>
        Date:   Wed Apr 24 19:11:28 2024 +0300

            [juniper_junos]: include log.file.device_id and log.file.inode in base-fields (elastic#9657)

            * fix(juniper_junos): include log.file.device_id and log.file.inode in base-fields.yml

            * fix(juniper_junos): update README.md
@elasticmachine
Copy link

elasticmachine commented May 1, 2024 •
edited

⏳ Build in-progress, with failures

Failed CI Steps

History

@milan-elastic
Copy link
Contributor Author

Raising another PR with better commit history so each packages can be reviewed with commits: #9768

@milan-elastic milan-elastic deleted the add-dashboard-level-filter branch May 1, 2024 10:36
@elastic-sonarqube Elastic SonarQube
Copy link

Quality Gate passed Quality Gate passed

Issues
0 New issues
0 Fixed issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarQube

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@milan-elastic @elasticmachine