Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sql Input Package #5167

Merged
merged 10 commits into from
Feb 15, 2023
Merged

Sql Input Package #5167

merged 10 commits into from
Feb 15, 2023

Conversation

ishleenk17
Copy link
Contributor

@ishleenk17 ishleenk17 commented Feb 2, 2023 •
edited
Loading

  • Enhancement

What does this PR do?

This PR creates the SQL Input Package which can be used to retrieve metrics by querying any SQL database.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

How to test this PR locally

Install the input package through Kibana. Bring up any SQL database. Specify the configurations as mentioned in the README. The metrics should be visible in Kibana

Related issues

Screenshots

Configuration for Input Package
Screenshot 2023-02-09 at 1 20 10 PM

Metrics Retrieval for Mysql database using SQL input Package

Screenshot 2023-02-09 at 1 21 23 PM

@ishleenk17 ishleenk17 requested a review from a team February 2, 2023 06:11
@ishleenk17 ishleenk17 self-assigned this Feb 2, 2023
@elasticmachine
Copy link

elasticmachine commented Feb 2, 2023 •
edited
Loading

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-02-15T07:13:15.795+0000

  • Duration: 13 min 42 sec

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

lalit-satapathy
lalit-satapathy reviewed Feb 6, 2023
View reviewed changes
Copy link
Collaborator

@lalit-satapathy lalit-satapathy left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An ideal scenario for generic sql input is driver is selected by a drop-down list and it provides a sample DSN notation for the driver pre-populated. May be our UI does not support such selections for now?

packages/sql_input/docs/README.md Show resolved Hide resolved
packages/sql_input/docs/README.md Outdated Show resolved Hide resolved
packages/sql_input/docs/README.md Outdated Show resolved Hide resolved
@ishleenk17 ishleenk17 marked this pull request as ready for review February 6, 2023 10:52
@agithomas
Copy link
Contributor

Can you please verify if the below test case is passed? This is because, i do not find any test cases attached with this PR.

  • Queries having special characters such as quotes, double quotes, special characters. Please verify if the query is fetching results.

Can there be preconditions / troubleshooting section added such as

  • Ensure that query does not end with semi colon

  • Ensure correctness of the query before executing

  • Ensure the database user executing the database query has the required permissions

  • For oracle, you can consider adding a link to the Oracle integration documentation to find out the pre-requisite. This includes setting of certain environment variables, installation of client

  • Recommend to change the text Hosts in Manifest.yml to Host / Connection String

agithomas
agithomas reviewed Feb 8, 2023
View reviewed changes
packages/sql_input/manifest.yml Show resolved Hide resolved
packages/sql_input/docs/README.md Outdated Show resolved Hide resolved
agithomas
agithomas reviewed Feb 8, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments added

@agithomas
Copy link
Contributor

In the screenshot, the name sql_ishleen is found. Please change it to a more appropriate name suiting the documentation bestpractices

@agithomas
Copy link
Contributor

Are there plans to include system testing and it's files & configs as part of this PR?

@ishleenk17
Copy link
Contributor Author

Can you please verify if the below test case is passed? This is because, i do not find any test cases attached with this PR.

  • Queries having special characters such as quotes, double quotes, special characters. Please verify if the query is fetching results.

Can there be preconditions / troubleshooting section added such as

  • Ensure that query does not end with semi colon
  • Ensure correctness of the query before executing
  • Ensure the database user executing the database query has the required permissions
  • For oracle, you can consider adding a link to the Oracle integration documentation to find out the pre-requisite. This includes setting of certain environment variables, installation of client
  • Recommend to change the text Hosts in Manifest.yml to Host / Connection String

This will basically use the SQL beats module. So if we are claiming support doe special characters for our DB's using SQL as input then that holds true for the input package. Whatever SQL module supports, SQL input package will support.

We can point out the specific DB's Integrations to provide the complete way of configuring and prerequisites. I don't think we should write details about all the individual DB's here. Linking to the specific DB readme would be better

@ishleenk17
Copy link
Contributor Author

Are there plans to include system testing and it's files & configs as part of this PR?

System Tests are in plan to be added for Input packages. They are not currently available.

agithomas
agithomas reviewed Feb 13, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review feedback shared.

packages/sql_input/docs/README.md Show resolved Hide resolved
@elastic elastic deleted a comment from lalit-satapathy Feb 14, 2023
@lalit-satapathy
Copy link
Collaborator

LGTM for the first release.

agithomas
agithomas reviewed Feb 15, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feedback shared.

packages/sql_input/docs/README.md
Expects a two-column table that looks like a key/value result. The left column is considered a key and the right column the value. This mode generates a single event on each fetch operation.

table:
Expects any number of columns. This mode generates a single event for each row.
Copy link
Contributor

@agithomas agithomas Feb 15, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It does not generate when query does not return any rows. So, suggest to clearly state this.

@agithomas
Copy link
Contributor

Suggestion: It may be best to replace or add additional screenshot that shows how to include multiple queries.

agithomas
agithomas reviewed Feb 15, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review comments updated

agithomas
agithomas approved these changes Feb 15, 2023
View reviewed changes
Copy link
Contributor

@agithomas agithomas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved

@ishleenk17 ishleenk17 merged commit b723d42 into elastic:main Feb 15, 2023
@elasticmachine
Copy link

Package sql - 0.0.1 containing this change is available at https://epr.elastic.co/search?package=sql

bhapas pushed a commit to bhapas/integrations that referenced this pull request Feb 16, 2023
@ishleenk17 @bhapas
Sql Input Package (elastic#5167)
163ca0f 
* Change cpu and memory calculation in vsphere

* SQL Input Package: Draft version

* Revert "Change cpu and memory calculation in vsphere"

This reverts commit 94487c3.

* Update Codeowners

* Update README

* Update README.md

* Update README.md

* Update README.md

* Update README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lalit-satapathy lalit-satapathy lalit-satapathy left review comments

@agithomas agithomas agithomas approved these changes

Assignees

@ishleenk17 ishleenk17

Labels
Integration:sql_input SQL Input New Integration
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Build sql input package
5 participants
@ishleenk17 @elasticmachine @agithomas @lalit-satapathy @andrewkroh