BeaconAPI Spec Conformance

[z3n-chada]

Describe the bug
There are spec conformance issues in the BeaconAPI json-rpc endpoint.

Expected behavior

API Path	Inputs	Issue
/eth/v1/beacon/states/(state)/root	"head"	No root key in the data field. No execution_optimistic field
/eth/v1/beacon/states/{state_id}/fork	"head"	No execution_optimistic field
/eth/v1/beacon/states/{state_id}/finality_checkpoints	"head"	No execution_optimistic field
/eth/v1/beacon/states/{state_id}/validators	"head"	No execution_optimistic field
/eth/v1/beacon/states/{state_id}/committees	"head"	No execution_optimistic field
/eth/v1/beacon/states/{state_id}/sync_committees	"head"	No execution_optimistic field
/eth/v2/beacon/blocks/{block_id}	"head"	No execution_optimistic field
/eth/v1/beacon/headers	"head"	No execution_optimistic field
/eth/v1/node/syncing	None	No is_optimistic field

Steps to Reproduce
Use the json-rpc endpoint.

Screenshots

Desktop (please complete the following information):

• OS: N/A
• Version: N/A
• Branch: master
• Commit hash: N/A

[dapplion]

Related to #3682

Tests should download the OpenAPI spec and assert responses follow its format

[dadepo]

I ended up reaching for Ajv to perform the validation of response object against the schema, but the problem is, the beacon-api spec does not have any of the properties of the response objects as required.

This makes it impossible (or convoluted) to perform a strict validation because a response object that misses any of the properties would still pass the validation check.

Does anyone know if it was a conscious design choice not marking the properties of the response object?

[dapplion]

@dadepo SSZ fromJson already does property validation for all important data structures on the ingress side at runtime

To ensure we conform with the spec on output, just generate testers from the OpenAPI schema and via tests check that we conform to that spec at test runtime

• Add good full coverage beacon API tests #3682

[dadepo]

just generate testers from the OpenAPI schema

@dapplion do you by chance have any tool in mind that does this? The ones I see are for either generating client libraries from the OpenAPI schema, or this paid tool that offers help with testing.

The only way I found to validate the response is to use a tool like Ajv to check that the response from the API conforms to the OpenAPI spec.

[dapplion]

just generate testers from the OpenAPI schema

@dapplion do you by chance have any tool in mind that does this? The ones I see are for either generating client libraries from the OpenAPI schema, or this paid tool that offers help with testing.

The only way I found to validate the response is to use a tool like Ajv to check that the response from the API conforms to the OpenAPI spec.

https://www.google.com/search?q=openapi+to+json+schema+converter

https://github.com/openapi-contrib/openapi-schema-to-json-schema

Using generated client libraries may also work but may be more complicated

[dadepo]

https://github.com/openapi-contrib/openapi-schema-to-json-schema

This does not generate testers though, but converts OpenAPI 3.0 to JSON Schema Draft 4. The output of the conversion, can't also be used directly for comparison in tests. As the tests needs to validate that the json returned by the API is the same as what is specified in the spec.

Using generated client libraries may also work but may be more complicated

I'll check this out but I am not sure how that will work, The client libraries will be generated client code that calls the API, they might not have facility to validate the response. And we do not need client code to call the api as we have that already in lodestar itself.

[dapplion]

• Since Check API definitions against OpenAPI spec #4493 Lodestar is now guaranteed to conform to the JSON schema from OpenAPI spec's definition at the commit set in
  

  lodestar/packages/api/test/unit/beacon/oapiSpec.test.ts

  Line 26 in 6a37cca

  const version = "v2.3.0";

Closing for now, tho note there is still work to do to ensure Lodestar logically conforms to the spec.