Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[APM] migrate to io-ts #42961

Merged
merged 11 commits into from Aug 21, 2019

Conversation

@dgieselaar
Copy link
Contributor

commented Aug 8, 2019

I've migrated some of our routes to io-ts, and with the help of some TypeScript magic and a quirky way of registering our routes, we now should have end-to-end runtime and type coverage without having to duplicate things for client and server.

Opening a draft PR to get some early feedback. I migrated the error and index pattern routes.

To summarize:

  • we replace joi with io-ts
  • instead of letting the routes register themselves, we export the route objects, import them in a root file and then register them at once via a router vessel. that router will inherit the types of the routes, and merge them all together.
  • we create route objects via a createRoute call. this allows us to get fully typed parameters in the request handler, inferred via the io-ts types.
  • the type of the router vessel is used to create a fully typed callApmApi method, that uses pathname and method as indexers to get inferred types for path, query and body params, and return types.

iots

@dgieselaar dgieselaar requested review from sqren and ogupte Aug 8, 2019


export type APICall<T extends ServerAPI<{}>> = T['call'];

const api: ServerAPI<{}> = {} as any;

This comment has been minimized.

Copy link
@dgieselaar

dgieselaar Aug 8, 2019

Author Contributor

this and below is just dummy stuff to test out the types.

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 8, 2019

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 12, 2019

@sqren We can also use this approach to generate the methods we now have in /rest, if we add a name property to the route description. Basically the TS transformation would look like:

{
  name: 'getErrorDistribution',
  path: `/api/apm/services/{serviceName}/errors/distribution`,
  method: 'GET',
  params: {
	path: t.type({
      serviceName: t.string
    }),
    query: t.intersection([
      t.partial({
        groupId: t.string
      }),
      defaultApiTypes
    ])
  },
  handler: async (req, params) => {
    ...
  }
}

to

const { client } = api;

client.getErrorDistribution({ serviceName, groupId, start, end, uiFilters })

And it would be fully typed.

Thoughts?

@sqren

This comment has been minimized.

Copy link
Member

commented Aug 12, 2019

const { client } = api;

client.getErrorDistribution({ serviceName, groupId, start, end, uiFilters })

And it would be fully typed.

Thoughts?

My knee-jerk reaction was that I did not like this. Where's the http method, the url, potentially headers if we want to add those?
But then I thought a little more about it, and I think I really like it. All the things I mentioned above are just implementation details. I like how we can avoid all the ceremony of REST, have the transport details declare in a single config, and then just worry about calling the endpoint with the right arguments.

Controversial idea: we could take a page from graphql's book and get rid of path, method and headers entirely (all requests go to a single endpoint). We could keep name to identify the request handler on the backend. This would determine which params the client had to pass.

Btw. I am too young to remember xml-rpc but I believe that is what we are reinventing. I love it! :D

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 13, 2019

@sqren maybe we should use /api/apm/${apiName} for debugging purposes? For network inspection, curl, postman etc I imagine semi-RESTful endpoints are still useful. I'm also wary of losing some benefits of conventions around GET/POST/PUT/DELETE etc. But it might also not be entirely clear to me what the value of one endpoint without headers etc would be.

@sqren

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

maybe we should use /api/apm/${apiName} for debugging purposes? For network inspection, curl, postman etc I imagine semi-RESTful endpoints are still useful. I'm also wary of losing some benefits of conventions around GET/POST/PUT/DELETE etc

I also don't think it would be a great idea. For one it becomes much harder to cache content if everything goes to the same endpoint.

But it might also not be entirely clear to me what the value of one endpoint without headers etc would be

The benefit would be that it would simplify the API. Right now a route is identified by both a name, a path and a method. If we decided on a single endpoint for all routes, any route could be identified purely from the name.

With the rpc style approach you suggested last (which I liked) we don't have a single identifier. Eg. from the client we call it like

client.getErrorDistribution({ serviceName, groupId, start, end, uiFilters })

So the unique identifier for the route is getErrorDistribution, right? But how would it be called via curl? It will be something completely different. There is no correlation between the name identifier and the method/path.

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 13, 2019

So the unique identifier for the route is getErrorDistribution, right? But how would it be called via curl? It will be something completely different. There is no correlation between the name identifier and the method/path.

I guess that's up to the implementation. Nothing is holding us back from using name as a suffix for path. Ie, { name: 'getErrorsDistribution' } would register a route for '/api/apm/getErrorsDistribution' (and we'd no longer have to define path).

@sqren

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

I guess that's up to the implementation. Nothing is holding us back from using name as a suffix for path. Ie, { name: 'getErrorsDistribution' } would register a route for '/api/apm/getErrorsDistribution' (and we'd no longer have to define path).

Apart from the odd camelCased paths we'd end up with I think I like this approach. What do you think?

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 13, 2019

Yeah, I like it. Maybe I can demo what I have in today's standup, I'd like to know what @ogupte thinks about it as well.

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 13, 2019

Thinking about this some more: can we import server code (specifically the route descriptions) for client code? I mean runtime code, not just types. If we can't, we have to consider that the client call that we make needs to know enough to construct a request object. Ie, it needs to know the pathname, the request method, and the parameters. So that would limit us to the following API I think:

[RouteName]: ( params:Params, method:HttpMethod = 'GET' ) => ReturnTypeOfRoute

So that means even if we only have a POST route for RouteName, we would still need to explicitly define 'POST' as a method, because the client needs to know that it should send a POST to that API.

(this also means that we still have to differentiate between path, query and body parameters).

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 13, 2019

It does seems like I'm not able to import server-side code. The errors are really vague, but I'm going to assume that it just doesn't work for the time being. That also means I have to implement the pattern that we discussed client.getErrorDistribution({}) with a Proxy:

export const calls: Client<APMAPI> = new Proxy(
  {},
  {
    get: (obj, prop) => {
      return ({
        query = {},
        body = {},
        method = 'GET'
      }) => {
        return callApi({
          method,
          pathname: `/api/apm/${prop.toString()}`,
          query,
          body: body ? JSON.stringify(body) : undefined
        }) as any;
      };
    }
  }
) as Client<APMAPI>;

We can also keep it simple and opt for client('getErrorDistribution', {}).

@sqren

This comment has been minimized.

Copy link
Member

commented Aug 13, 2019

It does seems like I'm not able to import server-side code.

You might already do this but just to make sure we are on the same page: Typescript imports (interfaces, types etc) should work across /public and /server. If you want to share actual runtime code between client and server it should live in /common (instead of /public or /server)

@dgieselaar dgieselaar force-pushed the dgieselaar:api-io-ts branch from ee7ef71 to 528c62e Aug 14, 2019

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 14, 2019

@dgieselaar dgieselaar force-pushed the dgieselaar:api-io-ts branch from 528c62e to e94b05a Aug 14, 2019

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 14, 2019

@dgieselaar dgieselaar force-pushed the dgieselaar:api-io-ts branch from e94b05a to ef02690 Aug 18, 2019

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 20, 2019

@dgieselaar dgieselaar force-pushed the dgieselaar:api-io-ts branch from 413012a to ec97a2a Aug 20, 2019

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 20, 2019

As discussed via Zoom, decided to create a transactionSampleRate runtime type that is shared between client and server. Rest of the payload runtime type is server only. The data structure that the AddSettingFlyout component uses for validation is different than the one used for posting data to the server (and the server receiving it). This limits reusability of the complete payload runtime type.

Added tests for the runtime types I created and callApmApi & createApi.

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 20, 2019

dgieselaar added some commits Aug 8, 2019

t.number.is,
(input, context) => {
const value = Number(input);
return value >= 0 && value <= 1 && Number(value.toFixed(3)) === value

This comment has been minimized.

Copy link
@sqren

sqren Aug 21, 2019

Member

Number(value.toFixed(3)) === value looks so much nicer than the one I suggested 👍

// add _debug query parameter to all routes
if (key === 'query') {
codec = t.intersection([codec, debugRt]);
}

This comment has been minimized.

Copy link
@sqren

sqren Aug 21, 2019

Member

You used to do:

              params: {
                query: params.query
                  ? t.intersection([params.query, debugRt])
                  : debugRt
              },

Which made sense to me. But with this change, if the route doesn't define a query will _debug still be added?

This comment has been minimized.

Copy link
@dgieselaar

dgieselaar Aug 21, 2019

Author Contributor

No, that's a good catch actually. The way it's written now none of the {query,path,body} parameters get validated if there's no param types. Let me fix that (and write a test for it). Thanks!

@dgieselaar dgieselaar force-pushed the dgieselaar:api-io-ts branch from ec97a2a to 0315862 Aug 21, 2019

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 21, 2019

@sqren

sqren approved these changes Aug 21, 2019

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 21, 2019

@dgieselaar

This comment has been minimized.

Copy link
Contributor Author

commented Aug 21, 2019

retest

@elasticmachine

This comment has been minimized.

Copy link

commented Aug 21, 2019

@dgieselaar dgieselaar merged commit 4d54b5f into elastic:master Aug 21, 2019

53 checks passed

API integration tests node scripts/functional_tests --config test/api_integration/config.js --bail --debug
Details
Browser tests yarn run grunt test:browser-ci
Details
Build kbn_tp_sample_panel_action yarn build
Details
CLA All commits in pull request signed
Details
Check core API changes node scripts/check_core_api_changes
Details
Check file casing node scripts/check_file_casing --quiet
Details
Check licenses node scripts/check_licenses --dev
Details
Firefox smoke test node scripts/functional_tests --bail --debug --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/kibana-firefoxSmoke/node/linux-immutable/install/kibana --include-tag smoke --config test/functional/config.firefox.js
Details
Functional tests / Group 1 yarn run grunt run:functionalTests_ciGroup1
Details
Functional tests / Group 10 yarn run grunt run:functionalTests_ciGroup10
Details
Functional tests / Group 11 yarn run grunt run:functionalTests_ciGroup11
Details
Functional tests / Group 12 yarn run grunt run:functionalTests_ciGroup12
Details
Functional tests / Group 2 yarn run grunt run:functionalTests_ciGroup2
Details
Functional tests / Group 3 yarn run grunt run:functionalTests_ciGroup3
Details
Functional tests / Group 4 yarn run grunt run:functionalTests_ciGroup4
Details
Functional tests / Group 5 yarn run grunt run:functionalTests_ciGroup5
Details
Functional tests / Group 6 yarn run grunt run:functionalTests_ciGroup6
Details
Functional tests / Group 7 yarn run grunt run:functionalTests_ciGroup7
Details
Functional tests / Group 8 yarn run grunt run:functionalTests_ciGroup8
Details
Functional tests / Group 9 yarn run grunt run:functionalTests_ciGroup9
Details
Internationalization check node scripts/i18n_check --ignore-missing
Details
Interpreter functional tests node scripts/functional_tests --config test/interpreter_functional/config.js --bail --debug --kibana-install-dir ./build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64
Details
Jest integration tests yarn run grunt test:jest_integration
Details
Jest tests yarn run grunt test:jest
Details
Kibana visual regression tests yarn run percy exec node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/kibana-visualRegression/node/linux-immutable/install/kibana --config test/visual_regression/config.ts
Details
Mocha tests node scripts/mocha
Details
Plugin functional tests node scripts/functional_tests --config test/plugin_functional/config.js --bail --debug --kibana-install-dir ./build/oss/kibana-8.0.0-SNAPSHOT-linux-x86_64
Details
Project tests yarn run grunt test:projects
Details
Type check node scripts/type_check
Details
TypeScript - all files belong to a TypeScript project node scripts/check_ts_projects
Details
Verify NOTICE.txt node scripts/notice --validate
Details
Verify dependency versions yarn run grunt verifyDependencyVersions
Details
X-Pack Chrome Functional tests / Group 1 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup1/node/linux-immutable/install/kibana --include-tag ciGroup1
Details
X-Pack Chrome Functional tests / Group 10 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup10/node/linux-immutable/install/kibana --include-tag ciGroup10
Details
X-Pack Chrome Functional tests / Group 2 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup2/node/linux-immutable/install/kibana --include-tag ciGroup2
Details
X-Pack Chrome Functional tests / Group 3 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup3/node/linux-immutable/install/kibana --include-tag ciGroup3
Details
X-Pack Chrome Functional tests / Group 4 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup4/node/linux-immutable/install/kibana --include-tag ciGroup4
Details
X-Pack Chrome Functional tests / Group 5 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup5/node/linux-immutable/install/kibana --include-tag ciGroup5
Details
X-Pack Chrome Functional tests / Group 6 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup6/node/linux-immutable/install/kibana --include-tag ciGroup6
Details
X-Pack Chrome Functional tests / Group 7 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup7/node/linux-immutable/install/kibana --include-tag ciGroup7
Details
X-Pack Chrome Functional tests / Group 8 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup8/node/linux-immutable/install/kibana --include-tag ciGroup8
Details
X-Pack Chrome Functional tests / Group 9 node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-ciGroup9/node/linux-immutable/install/kibana --include-tag ciGroup9
Details
X-Pack Jest node scripts/jest --ci --verbose
Details
X-Pack Mocha yarn test
Details
X-Pack SIEM cyclic dependency test node legacy/plugins/siem/scripts/check_circular_deps
Details
X-Pack firefox smoke test node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-firefoxSmoke/node/linux-immutable/install/kibana --include-tag smoke --config test/functional/config.firefox.js
Details
X-Pack visual regression tests yarn run percy exec node scripts/functional_tests --debug --bail --kibana-install-dir /var/lib/jenkins/workspace/elastic+kibana+pull-request/JOB/x-pack-visualRegression/node/linux-immutable/install/kibana --config test/visual_regression/config.js
Details
eslint node scripts/eslint --no-cache
Details
kibana-ci Build finished.
Details
percy/kibana Visual review automatically approved, no visual changes found.
Details
prbot:outdated run `node scripts/update_prs 42961` to update
prbot:release note labels
sasslint node scripts/sasslint
Details

@dgieselaar dgieselaar deleted the dgieselaar:api-io-ts branch Aug 21, 2019

dgieselaar added a commit to dgieselaar/kibana that referenced this pull request Aug 21, 2019

[APM] migrate to io-ts (elastic#42961)
* [APM] migrate to io-ts

* Migrate remaining routes to io-ts

* Infer response type for useFetcher()

* Review feedback

* Use createRangeType util

* Extract & test runtime types

* Simplify runtime types

* Tests for createApi and callApmApi

* Use more readable variable names in runtime types

* Remove UIFilters query param for API endpoints where it is not supported

* Fix issues w/ default parameters in create_api

dgieselaar added a commit that referenced this pull request Aug 21, 2019

[APM] migrate to io-ts (#42961) (#43674)
* [APM] migrate to io-ts

* Migrate remaining routes to io-ts

* Infer response type for useFetcher()

* Review feedback

* Use createRangeType util

* Extract & test runtime types

* Simplify runtime types

* Tests for createApi and callApmApi

* Use more readable variable names in runtime types

* Remove UIFilters query param for API endpoints where it is not supported

* Fix issues w/ default parameters in create_api
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.