Skip to content
This repository has been archived by the owner on Mar 11, 2021. It is now read-only.

List workitems #120

Merged
merged 12 commits into from
Aug 19, 2016
Merged

List workitems #120

merged 12 commits into from
Aug 19, 2016

Conversation

tsmaeder
Copy link
Contributor

@tsmaeder tsmaeder commented Aug 15, 2016

This pull request fixes #101

The PR implements a very simple "query by example" functionality where you can add two parameters to the url "api/workitem"

  • page: , implements paging. If only one value is given, it is taken as the length parameters
  • filter: you can pass in a json object string. Only work items matching (with '=') the field values given in the example json will be selected.
    Example:
    http://localhost:8080/api/workitem?filter={"system.owner": "tmaeder","Name": "First","Type": "1"}&page=1,2

The PR has a query parser, which produces expression trees, which will in turn be compiled for use with Gorm. While the query system architecture is in place, only a small amount of expressions is implemented (for example, only "=", no other comparisons). The intention is to extend the system as needed.


This change is Reviewable

@kwk
Copy link
Collaborator

kwk commented Aug 15, 2016

:lgtm:

Apart from a few questions that I have and some missing comments that would increase the readers understanding while reviewing.

Great job Thomas! This is quite some complex stuff!

This is one moment where I wish Go would not only have duck-typing but and explicit implements keyword.


Reviewed 11 of 11 files at r1.
Review status: all files reviewed at latest revision, 16 unresolved discussions.


workitem.go, line 78 [r1] (raw file):

      return nil, params[0], nil
  }
  return nil, 100, nil

Just a small issue: Can The 100 looks like a default limit but it is hard-coded here. Maybe over time we should find another place to handle defaults.


criteria/criteria.go, line 153 [r1] (raw file):

}

// And

I wonder if I can find out how operator precedence is handled here ;)


criteria/iterator.go, line 9 [r1] (raw file):

}

type postOrderIterator struct {

It might be obvious but I think it's worth mentioning that postOrderIterator implements the criteria.ExpressionVisitor interface.


criteria/iterator_test.go, line 13 [r1] (raw file):

  r := Literal(5)
  expr := Equals(l, r)
  expected := []Expression{l, r, expr}

Would be nice if there was a comment about depth-first and left-to-right-order. But no real issue. As somebody reviewing the test it's nice to know what somebody expects, you know?


criteria/iterator_test.go, line 32 [r1] (raw file):

  }
  IteratePostOrder(expr, recorder)
  expected = []Expression{l, r}

So expr is no longer part of the expected slice because we return false as soon as we hit expr == r and thereby stop the visiting, right?


models/expression_compiler.go, line 15 [r1] (raw file):

array of errors
I think you meant "slice of errors".


models/expression_compiler.go, line 74 [r1] (raw file):

  right := a.Right().Accept(c)
  if left != nil && right != nil {
      return "(" + left.(string) + " " + op + " " + right.(string) + ")"

Ah, so here operator precedence is implicitly modeled through parenthesis, right? I mean without parenthesis we would need a precedence defined but not with ( and ).

I wonder if a check needs to be implemented here:

if left != nil && right != nil {
    lstr, err := left.(string)
    if err != nil {
        return err
    }
    rstr, err := right.(string)
    if err != nil {
        return err
    }
    return "(" + lstr + " " + op + " " + rstr + ")"
}
return nil

Can one possible inject a ( or ) into the BinaryExpression to manipulate the query? Somwhat like SQL injection?


models/expression_compiler.go, line 90 [r1] (raw file):

func (c *expressionCompiler) Parameter(v *criteria.ParameterExpression) interface{} {
  c.parameterCount++
  return "?"

This one I don't get: Parameter. Can you please explain what ? means? I mean, I understand or and and.


models/expression_compiler.go, line 99 [r1] (raw file):

      if exp.Annotation(jsonAnnotation) == true {
          result = true
          return false

A comment would be nice that says that false stops the iteration.


models/expression_compiler.go, line 108 [r1] (raw file):

a->'foo'
What is the -> doing? I saw that in the expressionCompiler.Field function above but couldn't find out what it is.


models/expression_compiler.go, line 124 [r1] (raw file):

return "'"" + t + ""'"
To we need to clean t before adding the ' around it? I mean, is this a potential place to inject code?


models/expression_compiler_test.go, line 7 [r1] (raw file):

  "testing"

  . "github.com/almighty/almighty-core/criteria"

I'm no big fan of dot imports but I understand why one wants to do it here. Let's just not use it in other code that we write ourselves.


models/expression_compiler_test.go, line 16 [r1] (raw file):

func TestParameter(t *testing.T) {
  expect(t, And(Literal(true), Parameter()), "(true and ?)", 1)

Damn it, I still don't get the ? :D


models/expression_compiler_test.go, line 27 [r1] (raw file):

func expect(t *testing.T, expr Expression, expectedClause string, expectedParameters uint16) {
  clause, parameters, err := Compile(expr)

So this is where the slice of errors gets used, right? Is it used somewhere else outside of tests? I didn't notice it.


models/gorm_repository.go, line 178 [r1] (raw file):

  var rows []WorkItem
  db := r.ts.tx.Where(where)
  if start != nil {

If negative values for start and limit are not allowed we should either return with an error or ignore them. Will a negative offset fetch results from the end of the results?

What if start and limit overlap? Can this happen?


query/simple/simple_parser.go, line 12 [r1] (raw file):

true and attribute1=value1 and attribute2=value2

So the Literal(true) is always included? I only see the Literal(true) below if either the input byte string is empty or if the number of unmarshaled things is less than 1.


Comments from Reviewable

@kwk kwk added the 😴 REST label Aug 16, 2016
@tsmaeder
Copy link
Contributor Author

Review status: all files reviewed at latest revision, 16 unresolved discussions.


workitem.go, line 78 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Just a small issue: Can The 100 looks like a default limit but it is hard-coded here. Maybe over time we should find another place to handle defaults.

Yes, it looks odd to have a hard-coded default. However, I would expect that clients always pass in paging parameters for safety. Also, we'll have to think about hard-limiting the number of items returned items to prevent the server from blowing up. We'll have to return an indication that there are more items to fetch, in that case, though. All these concerns are not addressed right now.

criteria/criteria.go, line 153 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

I wonder if I can find out how operator precedence is handled here ;)

Operator precedence is only a problem with textual expressions. In an expression tree, the tree gives a defined precedence (in our case left-right, depth-first

criteria/iterator.go, line 9 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

It might be obvious but I think it's worth mentioning that postOrderIterator implements the criteria.ExpressionVisitor interface.

I think an optional "implements" keyword in go might make impelementation easier.

criteria/iterator_test.go, line 13 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Would be nice if there was a comment about depth-first and left-to-right-order. But no real issue. As somebody reviewing the test it's nice to know what somebody expects, you know?

To me, the code looks obvious, but then I wrote it;-) About comments in tests...In Java, I would make many small test methods that would explain their intent in their name. Not sure what to do in golang yet.

criteria/iterator_test.go, line 32 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

So expr is no longer part of the expected slice because we return false as soon as we hit expr == r and thereby stop the visiting, right?

Correct, this is about checking the early iteration cutoff.

models/expression_compiler.go, line 15 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

array of errors
I think you meant "slice of errors".

yes, of course.

models/expression_compiler.go, line 74 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Ah, so here operator precedence is implicitly modeled through parenthesis, right? I mean without parenthesis we would need a precedence defined but not with ( and ).

I wonder if a check needs to be implemented here:

if left != nil && right != nil {
    lstr, err := left.(string)
    if err != nil {
        return err
    }
    rstr, err := right.(string)
    if err != nil {
        return err
    }
    return "(" + lstr + " " + op + " " + rstr + ")"
}
return nil

Can one possible inject a ( or ) into the BinaryExpression to manipulate the query? Somwhat like SQL injection?

I don't think we need to check for the expressions returning string...this would be a programming error that I would "assert" against in java. As for the injection...I hide my face in shame. Typical rookie mistake :-( Will go stand in corner and then fix it.

models/expression_compiler.go, line 90 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

This one I don't get: Parameter. Can you please explain what ? means? I mean, I understand or and and.

Parameter as in sql parameter: "foo = :someParam" It's basically a free variable in the expression

models/expression_compiler.go, line 99 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

A comment would be nice that says that false stops the iteration.

In this case, I disagree: The semantics of the return value are documented in the comments for "IterateParents" and don't need to to be repeated here.

models/expression_compiler.go, line 108 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

a->'foo'
What is the -> doing? I saw that in the expressionCompiler.Field function above but couldn't find out what it is.

it's the postgres operator for referencing a field.

models/expression_compiler.go, line 124 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

return "'"" + t + ""'"
To we need to clean t before adding the ' around it? I mean, is this a potential place to inject code?

yes, this would allow for injection. But I believe the correct fix would be to not use literals, but parameters to avoid this.

models/expression_compiler_test.go, line 16 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Damn it, I still don't get the ? :D

I think sql parameters are the model to think of: if I parse the expression "where column1= :param1", :param1 would be the thing ending up as a parameter

models/expression_compiler_test.go, line 27 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

So this is where the slice of errors gets used, right? Is it used somewhere else outside of tests? I didn't notice it.

We are checking for nil when listing work items. https://github.com//issues/94 and https://github.com//issues/100 about returning structured errors from API calls. I it would make sense to address this at that time.

models/gorm_repository.go, line 178 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

If negative values for start and limit are not allowed we should either return with an error or ignore them. Will a negative offset fetch results from the end of the results?

What if start and limit overlap? Can this happen?

Negative numbers will be ignored.

query/simple/simple_parser.go, line 12 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

true and attribute1=value1 and attribute2=value2

So the Literal(true) is always included? I only see the Literal(true) below if either the input byte string is empty or if the number of unmarshaled things is less than 1.

Legacy comment. Fixed.

Comments from Reviewable

@kwk
Copy link
Collaborator

kwk commented Aug 17, 2016

Review status: all files reviewed at latest revision, 10 unresolved discussions, some commit checks failed.


workitem.go, line 78 [r1] (raw file):
Okay I see. So far I've only worked with one API that deals with "next"-links: The docker registry API (https://docs.docker.com/registry/spec/api/#/listing-image-tags). There they have mechanism to embed a "Link" attribute in the header:

Quoted 11 lines of code…

200 OK
Content-Type: application/json
Link: <<url>?n=<n from the request>&last=<last tag value from previous response>>; rel="next"

{
  "name": <name>,
  "tags": [
    <tag>,
    ...
  ]
}

To get the next result set, a client would issue the request as follows, using the value encoded in the RFC5988 Link header:

GET /v2/<name>/tags/list?n=<n from the request>&last=<last tag value from previous response>

Maybe it makes sense to look into the rfc5988.


criteria/iterator.go, line 9 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

I think an optional "implements" keyword in go might make impelementation easier.

Agreed. But for now the comment would be nice.

criteria/iterator_test.go, line 13 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

To me, the code looks obvious, but then I wrote it;-) About comments in tests...In Java, I would make many small test methods that would explain their intent in their name. Not sure what to do in golang yet.

I'm not sure that this is what you meant but in Go you can either write `TestXxx()` function where Xxx explains the intent or you can do **Subtests** if you don't want to repeat setup and tear down code. See this page: https://golang.org/pkg/testing/

models/expression_compiler.go, line 74 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

I don't think we need to check for the expressions returning string...this would be a programming error that I would "assert" against in java.
As for the injection...I hide my face in shame. Typical rookie mistake :-( Will go stand in corner and then fix it.

I don't get if you're just trying to be sarcastic or if you really mean it.

models/expression_compiler.go, line 90 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

Parameter as in sql parameter: "foo = :someParam" It's basically a free variable in the expression

Sorry Thomas but you basically explain parameter with parameter. That didn't help much :) Where's the difference to a field? Does a field always have to be a column name in the relational database and is a parameter something like a placeholder in prepared statements?

models/expression_compiler.go, line 99 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

In this case, I disagree: The semantics of the return value are documented in the comments for "IterateParents" and don't need to to be repeated here.

Repeating helps remembering and since 5 out of 9 lines have some `true` or `false` in them it would be nice to have a comment.

models/expression_compiler.go, line 124 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

yes, this would allow for injection. But I believe the correct fix would be to not use literals, but parameters to avoid this.

If one uses parameters, does one have to use prepares statements? It's not a bad thing, I guess, but I'm not sure if it allows for creative filtering with an arbitrary number of filter parameters.

models/gorm_repository.go, line 178 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

Negative numbers will be ignored.

How will they be ignored? I've digged deep into gorm and couldn't found a place.

Comments from Reviewable

@tsmaeder
Copy link
Contributor Author

Review status: all files reviewed at latest revision, 10 unresolved discussions, some commit checks failed.


workitem.go, line 78 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Okay I see. So far I've only worked with one API that deals with "next"-links: The docker registry API (https://docs.docker.com/registry/spec/api/#/listing-image-tags). There they have mechanism to embed a "Link" attribute in the header:

200 OK
Content-Type: application/json
Link: <<url>?n=<n from the request>&last=<last tag value from previous response>>; rel="next"

{
  "name": <name>,
  "tags": [
    <tag>,
    ...
  ]
}

To get the next result set, a client would issue the request as follows, using the value encoded in the RFC5988 Link header:

GET /v2/<name>/tags/list?n=<n from the request>&last=<last tag value from previous response>

Maybe it makes sense to look into the rfc5988.

I am following the guidelines from jsonapi.org.

criteria/iterator_test.go, line 13 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

I'm not sure that this is what you meant but in Go you can either write TestXxx() function where Xxx explains the intent or you can do Subtests if you don't want to repeat setup and tear down code. See this page: https://golang.org/pkg/testing/

The implementation is not the problem...I just don't have an opinion on what I consider good practice.

models/expression_compiler.go, line 74 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

I don't get if you're just trying to be sarcastic or if you really mean it.

Both, it's a stupid mistake with potentially disastrous results.

models/expression_compiler.go, line 90 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Sorry Thomas but you basically explain parameter with parameter. That didn't help much :) Where's the difference to a field? Does a field always have to be a column name in the relational database and is a parameter something like a placeholder in prepared statements?

When you prepare a statement in sql, you can have parameters in the statement. You can then repeatedly execute the same statement while passing in diffrerent values for the parameters. Mathematically, that is a "free variable" in the expression:

https://en.wikipedia.org/wiki/Free_variables_and_bound_variables


models/expression_compiler.go, line 124 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

If one uses parameters, does one have to use prepares statements? It's not a bad thing, I guess, but I'm not sure if it allows for creative filtering with an arbitrary number of filter parameters.

Preparing a statement is basically just parsing and constructing an execution plan. You will do this anyway for every statement. By calling "prepareStatement" or some such function, you're just reusing the result and you're giving the storage engine a better chance at reusing execution plan. We will construct a new sql statement on every call anyway, so the use of parameters does not limit the possibilities here.

models/gorm_repository.go, line 178 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

How will they be ignored? I've digged deep into gorm and couldn't found a place.

I tried it.

Comments from Reviewable

@kwk
Copy link
Collaborator

kwk commented Aug 17, 2016

:lgtm:


Review status: 6 of 11 files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.


models/expression_compiler.go, line 90 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

When you prepare a statement in sql, you can have parameters in the statement. You can then repeatedly execute the same statement while passing in diffrerent values for the parameters.
Mathematically, that is a "free variable" in the expression:

https://en.wikipedia.org/wiki/Free_variables_and_bound_variables

Okay I know prepared statements. I just wasn't sure if that's what we were talking about. But thanks.

models/expression_compiler.go, line 124 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

Preparing a statement is basically just parsing and constructing an execution plan. You will do this anyway for every statement. By calling "prepareStatement" or some such function, you're just reusing the result and you're giving the storage engine a better chance at reusing execution plan. We will construct a new sql statement on every call anyway, so the use of parameters does not limit the possibilities here.

Totally agree. Thanks

models/gorm_repository.go, line 178 [r1] (raw file):

Previously, tsmaeder (Thomas Mäder) wrote…

I tried it.

Haha, no that's not good enough :)

Comments from Reviewable

@kwk
Copy link
Collaborator

kwk commented Aug 17, 2016

Reviewed 5 of 5 files at r2.
Review status: all files reviewed at latest revision, all discussions resolved, some commit checks failed.


Comments from Reviewable

@aslakknutsen
Copy link
Contributor

:lgtm: Good stuff!


Comments from Reviewable

@tsmaeder
Copy link
Contributor Author

Review status: 6 of 12 files reviewed at latest revision, all discussions resolved.


models/gorm_repository.go, line 178 [r1] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Haha, no that's not good enough :)

Found it: in vendor/github.com/jinzhu/gorm/dialect_common.go#LimitAndOffsetSQL(...)

Comments from Reviewable

@kwk
Copy link
Collaborator

kwk commented Aug 18, 2016

:lgtm_strong:


Reviewed 6 of 6 files at r3.
Review status: all files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.


models/expression_compiler.go, line 70 [r3] (raw file):

Quoted 5 lines of code…

if strings.Contains(f.FieldName, "'") {
      // beware of injection, it's a reasonable restriction for field names, make sure it's not allowed when creating wi types
      c.err = append(c.err, fmt.Errorf("single quote not allowed in field name"))
      return nil
  }

I'm not sure if this prevents queries where you have artificial columns like

SELECT id as 'Project ID' FROM table WHERE Field->'Project ID' = 1;

I know this looks like some contrived example but think of a subqueries in the SELECT part where a new column name has to be used. Since we write the queries ourselves it shouldn't be a problem to simply avoid having spaces in our artificial columns and thereby get rid of single quotes.

Anyway, what about this Scope.Dialect().Quote(string) (string) method, can't we use that to pass in unsafe strings and get a safe string out? The PostgreSQL embeds the common dialect and that implements the Quote method here:

// documentation taken from Dialect:
// Quote quotes field name to avoid SQL parsing exceptions by using a reserved word as a field name
func (commonDialect) Quote(key string) string {
    return fmt.Sprintf(`"%s"`, key)
}

No this does what is says but it doesn't help to avoid SQL injections.

Another alternative would be to use the pq.QuoteIdentifier() function.

Sorry for this lengthy comment.


models/expression_compiler.go, line 99 [r3] (raw file):

func (c *expressionCompiler) Parameter(v *criteria.ParameterExpression) interface{} {
  c.err = append(c.err, fmt.Errorf("Parameter expression not supported"))

How come this is no longer supported?


models/expression_compiler.go, line 157 [r3] (raw file):

}

func wrapJSON(isJSON bool, value string) string {

Is wrapJSON used at all now?


Comments from Reviewable

@tsmaeder
Copy link
Contributor Author

Review status: all files reviewed at latest revision, 3 unresolved discussions, some commit checks failed.


models/expression_compiler.go, line 70 [r3] (raw file):

Previously, kwk (Konrad Kleine) wrote…

if strings.Contains(f.FieldName, "'") {
        // beware of injection, it's a reasonable restriction for field names, make sure it's not allowed when creating wi types
        c.err = append(c.err, fmt.Errorf("single quote not allowed in field name"))
        return nil
    }

I'm not sure if this prevents queries where you have artificial columns like

SELECT id as 'Project ID' FROM table WHERE Field->'Project ID' = 1;

I know this looks like some contrived example but think of a subqueries in the SELECT part where a new column name has to be used. Since we write the queries ourselves it shouldn't be a problem to simply avoid having spaces in our artificial columns and thereby get rid of single quotes.

Anyway, what about this Scope.Dialect().Quote(string) (string) method, can't we use that to pass in unsafe strings and get a safe string out? The PostgreSQL embeds the common dialect and that implements the Quote method here:

// documentation taken from Dialect:
// Quote quotes field name to avoid SQL parsing exceptions by using a reserved word as a field name
func (commonDialect) Quote(key string) string {
  return fmt.Sprintf(`"%s"`, key)
}

No this does what is says but it doesn't help to avoid SQL injections.

Another alternative would be to use the pq.QuoteIdentifier() function.

Sorry for this lengthy comment.

This restriction only applies to field identifiers of fields stored in JSON, where the field names need to be single quoted anyway. These are NOT sql column names.

models/expression_compiler.go, line 99 [r3] (raw file):

Previously, kwk (Konrad Kleine) wrote…

How come this is no longer supported?

because we're not longer using it for executing the queries. No need.

models/expression_compiler.go, line 157 [r3] (raw file):

Previously, kwk (Konrad Kleine) wrote…

Is wrapJSON used at all now?

Nope

Comments from Reviewable

@kwk
Copy link
Collaborator

kwk commented Aug 19, 2016

:lgtm:


Reviewed 1 of 1 files at r4.
Review status: all files reviewed at latest revision, all discussions resolved.


Comments from Reviewable

@kwk kwk merged commit de975dc into fabric8-services:master Aug 19, 2016
}

if len(result) != 1 {
t.Errorf("unexpected length, is %d but should be %d", 1, len(result))
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just found that the arguments to the Errorf call should be switched, right?

t.Errorf("unexpected length, is %d but should be %d", 1, len(result))

should be

t.Errorf("unexpected length, is %d but should be %d", len(result), 1)

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

As a User I should be able to List created WorkItems
3 participants