Skip to content
This repository has been archived by the owner on Oct 25, 2024. It is now read-only.

enhancement: add filtering to GraphQL API queries #721

Conversation

deekerno
Copy link
Contributor

@deekerno deekerno commented Apr 1, 2023

Closes #637.

Changelog

  • Add arguments module to fuel-indexer-schema::db; this is where all functionality for GraphQL query arguments should be located
  • Add pub fn parse_arguments(..); this is the entry point for parsing any query args
    -- todo!() stubs have been added for sorting and pagination
  • Add functionality and tests for the following filter types:
    -- id: select a record by its ID
    -- and / or: combine two filters together
    -- not: negate the behavior of a filter
    -- has: select records that have non-null values for columns in list
    -- in: select records that have a value matching an element in set
    -- gt / gte / lt / lte (greater than, etc.): select records whose value satisfies the comparison condition
    -- between: select records whose value falls between the bounds of the comparison condition
  • Move GraphQL API query to DB query functionality from fuel-indexer-schema/db/lib.rs to fuel-indexer-schema/db/queries.rs
  • Add additional variants to GraphQLError

Testing Plan

Added several new tests that check each element in a set of results after filtering as well as ensuring that there are no extra elements that do not satisfy the filter conditions. CI should pass.

Manual Testing

  1. Clear your database and ensure that WASM modules are up-to-date.
  2. Start the explorer indexer example with the beta-3 testnet:
cargo run --bin fuel-indexer -- run --fuel-node-host beta-3.fuel.network --fuel-node-port 80 --manifest examples/block-explorer/explorer-indexer/explorer_indexer.manifest.yaml
  1. Open the playground for the example: http://localhost:29987/api/playground/fuel_examples/explorer_indexer.
  2. Run the following query; notice that there are many results (this is the non-filtered example):
query {
  tx {
    id
    timestamp
    value
    hash
    status
    block {
    	id
    	height
    	timestamp
        hash
    }
  }
}
  1. Run the following query; ensure that there are three results returned (this example shows multiple filters on one entity [comprised of a complex comparison and simple comparison, which are joined together by a logical operator]).
query {
  tx {
    id
    timestamp
    value
    hash
    status
    block(filter: { height: { between: { min: 10, max: 15 } }, and: { timestamp: { lt: 1678484680 } } } ) {
    	id
    	height
    	timestamp
        hash
    }
  }
}
  1. Run the following query; ensure that there are now only two results returned (this example shows that multiple entities can be filtered in the same query [this is also an example for the membership filter]):
query {
  tx(filter: { hash: { in: ["adbf1b802450b4782358ff05661ef784a1c6ed47ad7b0cca645ff8fb75debb26", "9c6db9e9d75c2b357931ff7055af2da612a4017ccfe1af71b70ca88c22c9e2e3"] } } ) {
    id
    timestamp
    value
    hash
    status
    block(filter: { height: { between: { min: 10, max: 15 } }, and: { timestamp: { lt: 1678484680 } } } ) {
    	id
    	height
    	timestamp
        hash
    }
  }
}

Notes

Documentation updates will come in a separate PR so that this one doesn't get even bigger.

This implementation uses a fork of graphql-parser. The original version of the crate does not allow for parsing numbers greater than i64::MAX (as its internal number type is hard-coded to an i64); we allow for storing u64 and u128, so we need to figure out a way to parse u64s and u128s from valid GraphQL queries. As such, I've forked graphql-parser and added the necessary functionality; you can find the fork here and the diff for the changes here; it should be noted that the original crate has not released a new version in quite some time so there should be minimal burden from upstream changes.

@deekerno deekerno self-assigned this Apr 1, 2023
@deekerno deekerno force-pushed the deekerno/637-add-filtering-to-graphql-queries branch 6 times, most recently from 09d0762 to 88bb290 Compare April 7, 2023 04:41
@deekerno deekerno marked this pull request as ready for review April 7, 2023 06:05
@deekerno deekerno requested a review from a team April 7, 2023 06:05
@deekerno deekerno force-pushed the deekerno/637-add-filtering-to-graphql-queries branch 5 times, most recently from 94f4aba to b6dc578 Compare April 8, 2023 04:38
@deekerno deekerno changed the base branch from master to the-graphql-update April 10, 2023 14:42
@deekerno deekerno changed the base branch from the-graphql-update to feature/the-graphql-update April 10, 2023 15:13
@deekerno deekerno force-pushed the deekerno/637-add-filtering-to-graphql-queries branch from b6dc578 to 7605c11 Compare April 10, 2023 15:18
Copy link
Contributor

@ra0x3 ra0x3 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@deekerno

  • Fine to merge into the feature branch
  • As mentioned, I don't think we'll be able to really see if it works until we just start using it slash open it up
  • Just left comments regarding needing Rust docs and comments, as I think this GraphQL stuff has the most business logic of any part of the codebase (e.g., business logic being something that's not self documenting like everything else).

*elem = format!("{elem} AND {join_condition}")
}
} else {
joins.push(format!(
Copy link
Contributor

@ra0x3 ra0x3 Apr 10, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to abstract away some of this raw string usage? E.g.,

struct Join {
  rk_table: String,
  rk_col: String,
  pk_table: String,
  pk_col: String,
  is_the_condition: bool,
}



impl ToString for Join {
  fn to_string(&self) -> String {
    if self.is_the_condition {
        format!("{{}.{} = {}.{}", self.rk_table, self.rk_col, self.pk_table, self.pk_col)
    } else {
      // So on and so forth
    }
  }
}

I think making some of this a bit more Rust-y would make it easier to follow without changing any functionality/logic . Totally fine if not, just thinking how to make it a bit more readable

elements_string, self.namespace_identifier, self.entity_name,
);

if !joins.is_empty() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

All of this business logic here is gonna need a comment as it's not clear what's going on

}
}

// Set a nested JSON object as the value for this entity field.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think this comment helps me understand what this logic is doing

}
}

fn parse_query_elements(&self, db_type: &DbType) -> Vec<String> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It would be helpful when you add rust-doc comments to explain what strategy you're using to parse these query elements. The more info the better

}

impl UserQuery {
pub fn to_sql(&mut self, db_type: &DbType) -> String {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When you add rust-docs it'd be nice to know what the overall strategy is here for this to_sql method. The more info the better.

elements
}

fn get_topologically_sorted_joins(&mut self) -> Vec<JoinCondition> {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could use a helpful high-level rust doc comment here as well. The more info the better

@deekerno
Copy link
Contributor Author

Noted on the comments. Look out for a small refactoring/commenting PR to make this clearer.

@deekerno deekerno merged commit 925b13b into feature/the-graphql-update Apr 10, 2023
@deekerno deekerno deleted the deekerno/637-add-filtering-to-graphql-queries branch April 10, 2023 16:11
deekerno added a commit that referenced this pull request Apr 12, 2023
Add filtering functionality to GraphQL API; add E2E tests
deekerno added a commit that referenced this pull request Apr 14, 2023
Add filtering functionality to GraphQL API; add E2E tests
ra0x3 pushed a commit that referenced this pull request Apr 20, 2023
Add filtering functionality to GraphQL API; add E2E tests
deekerno added a commit that referenced this pull request Apr 23, 2023
Add filtering functionality to GraphQL API; add E2E tests
deekerno added a commit that referenced this pull request Apr 26, 2023
#763)

* enhancement: add filtering to GraphQL API queries (#721)

Add filtering functionality to GraphQL API; add E2E tests

* enhancement: add sorting and offset-based pagination to GraphQL API queries (#755)

* Add comments; rename to clearer variables

* Add sorting and offset-based pagination

* enhancement: add `PageInfo` object to paginated GraphQL responses (#761)

* Add PageInfo object to queries with a limit

* Adjust tests

* Change API response to snake_case

* docs: add docs for GraphQL filtering and pagination (#768)

* Add docs for GraphQL filtering and pagination

* Move QueryResponse to models module

* Address rvmelkonian feedback

* Address ra0x3 feedback

* rebase origin/master

* Add fuel-indexer-graphql(-parser) packages

* Fix index plugins failure

* Clippy fixes

---------

Co-authored-by: Alexander <alex@deekerno.com>
Co-authored-by: Rashad Alston <rashad@Rashads-Air.lan>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add filtering to GraphQL API queries
2 participants