Skip to content

[full-ci] enhancement: add support for natural language kql date ranges#7263

Merged
fschade merged 11 commits intomasterfrom
human-date-time-tokens
Sep 15, 2023
Merged

[full-ci] enhancement: add support for natural language kql date ranges#7263
fschade merged 11 commits intomasterfrom
human-date-time-tokens

Conversation

@fschade
Copy link
Copy Markdown
Contributor

@fschade fschade commented Sep 11, 2023

Description

follow up pr for the ocis kql integration, as the spec describes kql is able to select date ranges by writing natural language.

The following ranges are described in the spec and therefor supported by the ocis implementation:

  • Mtime:today - range: start of today till end of today
  • Mtime:yesterday - range: start of yesterday till end of yesterday
  • Mtime:"this week" - range: start of this week till end of this week
  • Mtime:"this month" - range: start of this month till end of this month
  • Mtime:"last month" - range: start of last month till end of last month
  • Mtime:"this year" - range: start of this year till end of this year
  • Mtime:"last year" - range: start of last year till end of last year

the ast will extended as follow:

	&ast.Ast{
		Nodes: []ast.Node{
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: ">="},
				Value:    mustParseTime(t, "2023-09-10"),
			},
			&ast.OperatorNode{Value: kql.BoolAND},
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: "<="},
				Value:    mustParseTime(t, "2023-09-10 23:59:59.999999999"),
			},
		},
	},
	&ast.Ast{
		Nodes: []ast.Node{
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: ">="},
				Value:    mustParseTime(t, "2023-09-09"),
			},
			&ast.OperatorNode{Value: kql.BoolAND},
			&ast.DateTimeNode{
				Key:      "Mtime",
				Operator: &ast.OperatorNode{Value: "<="},
				Value:    mustParseTime(t, "2023-09-09 23:59:59.999999999"),
			},
		},
	},

and so on....

following queries are now exactly covered as described in the spec:

* 2.1.2 AND Operator 
* 2.1.6 NOT Operator
* 2.1.8 OR Operator
* 2.1.12 Parentheses
* 2.3.5 Date Tokens
* 3.1.11 Implicit Operator
* 3.1.12 Parentheses
* 3.1.2 AND Operator
* 3.1.6 NOT Operator
* 3.1.8 OR Operator
* 3.2.3 Implicit Operator for Property Restriction
* 3.3.1.1.1 Implicit AND Operator
* 3.3.5 Date Tokens

beside that it also includes:

  • structured table tests

Related Issue

Motivation and Context

implement the spec

How Has This Been Tested?

  • unit tests
  • acceptance tests
  • e2e tests

Screenshots (if appropriate):

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)
  • Technical debt
  • Tests only (no source changes)

Checklist:

  • Code changes
  • Unit tests added
  • Acceptance tests added
  • Documentation ticket raised:

@fschade fschade force-pushed the human-date-time-tokens branch from 859045b to 9ac725c Compare September 11, 2023 14:36
@fschade fschade marked this pull request as ready for review September 15, 2023 07:28
@fschade fschade force-pushed the human-date-time-tokens branch from 01399ac to 6eea049 Compare September 15, 2023 07:31
@sonarqubecloud
Copy link
Copy Markdown

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 7 Code Smells

88.7% 88.7% Coverage
0.0% 0.0% Duplication

@fschade fschade merged commit 0f2b2b9 into master Sep 15, 2023
@delete-merged-branch delete-merged-branch bot deleted the human-date-time-tokens branch September 15, 2023 09:31
ownclouders pushed a commit that referenced this pull request Sep 15, 2023
…es (#7263)

* enhancement: add more kql spec tests and simplify ast normalization

* enhancement: kql parser error if query starts with AND

* enhancement: add kql docs and support for date and time only dateTimeRestriction queries

* enhancement: add the ability to decide how kql nodes get connected

connecting nodes (with edges) seem straight forward when not using group, the default connection for nodes with the same node is always OR. THis only applies for first level nodes, for grouped nodes it is defined differently. The KQL docs are saying, nodes inside a grouped node, with the same key are connected by a AND edge.

* enhancement: explicit error handling for falsy group nodes and queries with leading binary operator

* enhancement: use optimized grammar for kql parser and toolify pigeon

* enhancement: simplify error handling

* fix: kql implicit 'AND' and 'OR' follows the ms html spec instead of the pdf spec

* enhancement: add support for natural language kql date queries

* enhancement: structure kql parser tests into logical clusters

* fix: time-range error naming
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants