Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Elasticsearch: Support for visualizing logs in Explore #17605

Merged
merged 14 commits into from Jun 24, 2019

Conversation

@marefr
Copy link
Member

marefr commented Jun 17, 2019

What this PR does / why we need it:
Support for visualizing logs in Explore using Elasticsearch.

Left to do for MVP:

  • Use field settings from datasource settings
  • Figure out the best way of sending query from query field to datasource.
  • Replace context property with something that says that the query is a logs query.
  • Logs query should be run when switching to logs mode even though it's empty.
  • Try use the QueryField without any highlighting or auto-complete to get styling and ctrl+Return/Return to run query. Need to look into why the input field is lagging when input text to it. If lag cannot be removed, lets go for a regular html input field instead with Return handler.
  • Add types for elasticsearch options to datasource in similar manner as Influx datasource

Which issue(s) this PR fixes:
Closes #16812

Special notes for your reviewer:

@marefr marefr added this to Under review in Observability via automation Jun 17, 2019
@davkal davkal added this to the 6.3 milestone Jun 18, 2019
@davkal

This comment has been minimized.

Copy link
Contributor

davkal commented Jun 18, 2019

Reduce scope for MVP:

  • keep HTML input field for now, with Return handler
  • clean up query handling, ambiguous query handling in the datasource, might need to probe on query type
  • use field settings from datasource settings
  • make sure empty (all) queries work, perhaps via *

6.3 beta is blocked by this.

@kaydelaney

This comment has been minimized.

Copy link
Contributor

kaydelaney commented Jun 19, 2019

Re: Logs query should be run when switching to logs mode even though it's empty.

The simplest way to do this would probably be to add a dummy parameter to ElasticsearchQuery, that way when hasNonEmptyQuery() is called it fits the definition of being non-empty despite the query itself being an empty string. ("A target is non-empty when it has keys (with non-empty values) other than refId, key and context.")
Perhaps a bit hacky though

@kaydelaney kaydelaney force-pushed the 16812_es_explore branch from 79da520 to c66fc53 Jun 19, 2019
Copy link
Member Author

marefr left a comment

Typings of options looks great! Left some comments. One thing to consider is to copy the typings for the query function from DataSourceApi<> which now will give you typed queries/targets and proper Promise as return value.

@marefr

This comment has been minimized.

Copy link
Member Author

marefr commented Jun 19, 2019

Re: Logs query should be run when switching to logs mode even though it's empty.

The simplest way to do this would probably be to add a dummy parameter to ElasticsearchQuery, that way when hasNonEmptyQuery() is called it fits the definition of being non-empty despite the query itself being an empty string. ("A target is non-empty when it has keys (with non-empty values) other than refId, key and context.")
Perhaps a bit hacky though

@kaydelaney if we have a property on ElasticsearchQuery that says whether query is a logs query or not, that property can be added to hasNonEmptyQuery. What do you think?

@kaydelaney

This comment has been minimized.

Copy link
Contributor

kaydelaney commented Jun 19, 2019

Re: Logs query should be run when switching to logs mode even though it's empty.
The simplest way to do this would probably be to add a dummy parameter to ElasticsearchQuery, that way when hasNonEmptyQuery() is called it fits the definition of being non-empty despite the query itself being an empty string. ("A target is non-empty when it has keys (with non-empty values) other than refId, key and context.")
Perhaps a bit hacky though

@kaydelaney if we have a property on ElasticsearchQuery that says whether query is a logs query or not, that property can be added to hasNonEmptyQuery. What do you think?

Sounds good. 👍

@kaydelaney kaydelaney force-pushed the 16812_es_explore branch from c66fc53 to b8b9723 Jun 19, 2019
Copy link
Member Author

marefr left a comment

Great progress 👍 Left some comments that needs to be fixed.

@kaydelaney kaydelaney force-pushed the 16812_es_explore branch from 0740191 to 664e716 Jun 20, 2019
@marefr marefr marked this pull request as ready for review Jun 20, 2019
Copy link
Member Author

marefr left a comment

Changes looks great. Added some comments with request for minor code changes and cleanup.

Two things that doesn't work as expected is using the Clear All and clear query field row. They are actually clearing current query correctly, but the query isn't run at all which is the first problem.
Second, if you after clear click Run Query nothing happens.

Let me know if you'll need more information provided to reproduce these problems.

Copy link
Member Author

marefr left a comment

All features now works as expected 👍 Some minor cleanup and adding some tests would be good before we merge this, see comments..

@kaydelaney kaydelaney force-pushed the 16812_es_explore branch from c90bd39 to 3ebb4ad Jun 24, 2019
Copy link
Member Author

marefr left a comment

Great work! Approved. Since I'm the author I cannot approve this PR. @kaydelaney you have to approve it

@kaydelaney kaydelaney merged commit eecd8d1 into master Jun 24, 2019
2 checks passed
2 checks passed
build-branches-and-prs Workflow: build-branches-and-prs
Details
license/cla Contributor License Agreement is signed.
Details
Observability automation moved this from Under review to Done Jun 24, 2019
@kaydelaney kaydelaney deleted the 16812_es_explore branch Jun 24, 2019
ryantxu added a commit to ryantxu/grafana that referenced this pull request Jun 24, 2019
…-mapping-to-field

* grafana/master:
  Elasticsearch: Visualize logs in Explore (grafana#17605)
  Grafana-CLI: Wrapper for `grafana-cli` within RPM/DEB packages and config/homepath are now global flags (grafana#17695)
  Add guidelines for SQL date comparisons (grafana#17732)
  Docs: clarified usage of go get and go mod (grafana#17637)
  Project: Issue triage doc improvement (grafana#17709)
  Improvement: Grafana release process minor improvements (grafana#17661)
  TimePicker: New time picker dropdown & custom range UI (grafana#16811)
  RemoteCache: redis connection string parsing test (grafana#17702)
  Fix link in pkg/README (grafana#17714)
  Dashboard: Use Explore's Prometheus editor in dashboard panel edit (grafana#15364)
  Settings: Fix typo in defaults.ini (grafana#17707)
  Project: Adds a security policy (grafana#17698)
  Project: Adds support resource docs (grafana#17699)
  Document issue triage process (grafana#17669)
  noImplicitAny: slate (grafana#17681)
  config: fix connstr for remote_cache (grafana#17675)
  Explore: Improves performance of Logs element by limiting re-rendering (grafana#17685)
  Docs: Flag serve_from_sub_path as available in 6.3 (grafana#17674)
  @grafana/runtime: expose config and loadPluginCss (grafana#17655)
  noImplicitAny: Fix basic errors (grafana#17668)
ryantxu added a commit to ryantxu/grafana that referenced this pull request Jun 25, 2019
* grafana/master:
  Elasticsearch: Visualize logs in Explore (grafana#17605)
  Grafana-CLI: Wrapper for `grafana-cli` within RPM/DEB packages and config/homepath are now global flags (grafana#17695)
  Add guidelines for SQL date comparisons (grafana#17732)
  Docs: clarified usage of go get and go mod (grafana#17637)
  Project: Issue triage doc improvement (grafana#17709)
  Improvement: Grafana release process minor improvements (grafana#17661)
  TimePicker: New time picker dropdown & custom range UI (grafana#16811)
  RemoteCache: redis connection string parsing test (grafana#17702)
  Fix link in pkg/README (grafana#17714)
  Dashboard: Use Explore's Prometheus editor in dashboard panel edit (grafana#15364)
  Settings: Fix typo in defaults.ini (grafana#17707)
  Project: Adds a security policy (grafana#17698)
  Project: Adds support resource docs (grafana#17699)
  Document issue triage process (grafana#17669)
  noImplicitAny: slate (grafana#17681)
  config: fix connstr for remote_cache (grafana#17675)
  Explore: Improves performance of Logs element by limiting re-rendering (grafana#17685)
  Docs: Flag serve_from_sub_path as available in 6.3 (grafana#17674)
ryantxu added a commit to ryantxu/grafana that referenced this pull request Jun 25, 2019
* grafana/master: (33 commits)
  API: get list of users with additional auth info (grafana#17305)
  TimePicker: fixed minor issues with new timepicker (grafana#17756)
  Explore: Parses and updates TimeSrv in one place in Explore (grafana#17677)
  @grafana/ui: release (grafana#17754)
  Password: Remove PasswordStrength (grafana#17750)
  Devenv:SAML: devenv block with saml test app (grafana#17733)
  LDAP:Docs: add information on LDAP sync feature and update LDAP sync default (grafana#17689)
  Graph: Add data links feature (click on graph) (grafana#17267)
  Explore: Changes LogsContainer from a PureComponent to a Component (grafana#17741)
  Chore: Remove tether and tether drop dependency in grafana/ui (grafana#17745)
  noImplicitAny: time region manager etc. (grafana#17729)
  Panel: Fully escape html in drilldown links (was only sanitized before)  (grafana#17731)
  Alerting: Improve alert rule testing (grafana#16286)
  Elasticsearch: Visualize logs in Explore (grafana#17605)
  Grafana-CLI: Wrapper for `grafana-cli` within RPM/DEB packages and config/homepath are now global flags (grafana#17695)
  Add guidelines for SQL date comparisons (grafana#17732)
  Docs: clarified usage of go get and go mod (grafana#17637)
  Project: Issue triage doc improvement (grafana#17709)
  Improvement: Grafana release process minor improvements (grafana#17661)
  TimePicker: New time picker dropdown & custom range UI (grafana#16811)
  ...
ryantxu added a commit to ryantxu/grafana that referenced this pull request Jun 25, 2019
* grafana/master: (64 commits)
  release: update latest.json to v6.2.5 (grafana#17767)
  release: 6.2.5 changelog (grafana#17766)
  Fix typo s/Applicaiton/Application/ in error messages (grafana#17765)
  UserAdmin: UI for disabling users (grafana#17333)
  API: get list of users with additional auth info (grafana#17305)
  TimePicker: fixed minor issues with new timepicker (grafana#17756)
  Explore: Parses and updates TimeSrv in one place in Explore (grafana#17677)
  @grafana/ui: release (grafana#17754)
  Password: Remove PasswordStrength (grafana#17750)
  Devenv:SAML: devenv block with saml test app (grafana#17733)
  LDAP:Docs: add information on LDAP sync feature and update LDAP sync default (grafana#17689)
  Graph: Add data links feature (click on graph) (grafana#17267)
  Explore: Changes LogsContainer from a PureComponent to a Component (grafana#17741)
  Chore: Remove tether and tether drop dependency in grafana/ui (grafana#17745)
  noImplicitAny: time region manager etc. (grafana#17729)
  Panel: Fully escape html in drilldown links (was only sanitized before)  (grafana#17731)
  Alerting: Improve alert rule testing (grafana#16286)
  Elasticsearch: Visualize logs in Explore (grafana#17605)
  Grafana-CLI: Wrapper for `grafana-cli` within RPM/DEB packages and config/homepath are now global flags (grafana#17695)
  Add guidelines for SQL date comparisons (grafana#17732)
  ...
@marefr marefr changed the title Elasticsearch: Visualize logs in Explore Elasticsearch: Support for visualizing logs in Explore Jul 11, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
Observability
  
Done
3 participants
You can’t perform that action at this time.