Refactor API calls to be more future-proof

[vikramrajkumar]

From @theoreticalbts on August 18, 2015 16:21

Implementation of cryptonomex/graphene#236 has revealed a general issue when extending API calls. When adding parameters to an API call, it has been noted that we should have a cycle of introducing a new call, deprecating the old call, waiting for web UI code to be updated, and then removing the old call.

The deprecation cycle is necessary because adding parameters to an API call breaks every API client that uses that call, even if the new parameters have default values, because the FC reflection API does not understand how to interact with default values due to underlying deficiencies in C++

In this ticket, I propose a solution to this problem: Most or all API calls should be rewritten to take a single struct, containing the values that were previously passed as separate arguments. Defaults can then be placed in the C++ struct definition, and if some fields do not occur in the JSON dictionary provided by the client, then those fields should not be assigned by the API framework and thus will have their default values.

I should test that the API framework does in fact give fields that do not exist in the JSON object their default values.

Copied from original issue: cryptonomex/graphene#245

[vikramrajkumar]

From @theoreticalbts on August 18, 2015 16:33

We could even implement this API entirely in C++ without breaking clients that use positional arguments. Here's how I'd do it:

• Create a new API method macro called e.g. FC_EXTAPI
• FC_EXTAPI has the following semantics:
• If a single JSON object is passed, then the method is called directly with the object. Fields that exist in the C++ struct but not in the JSON object should never be assigned by the API framework, and thus will take their default values as specified in the C++ struct definition.
• If a single JSON value of a non-object type is passed, or a number of JSON entities other than 1 is passed, then the struct fields are filled in order with the provided values. If the provided values are fewer than the number of struct fields, then the remaining fields should not be assigned by the API framework, and thus will take their default values as specified in the C++ struct definition.

By straightforwardly defining parameter structs, we can replace FC_API methods with FC_EXTAPI methods, while maintaining complete backward compatibility from the client's point of view.

The win here is that, by maintaining support for positional arguments, the transition from brittle argument-based methods to extensible struct-based methods can occur without breaking client code.

[vikramrajkumar]

From @theoreticalbts on November 18, 2015 16:44

This should wait until cryptonomex/graphene#422 .

[jmjatlanta]

What about defining the defaults in the variadic template instead of the class? Similar to:

https://baptiste-wicht.com/posts/2015/03/named-optional-template-parameters-compile-time.html

It would basically be building the class at compile time using templates. The classes in graphene/app/api.hpp would have to be replaced with a "templated language". Readability would also suffer (IMO). But the gain is defaulted, named parameters.

I haven't walked this all the way through, and I have to say I haven't been deep in templates for a bit. I can't say I even like them. But it does seem to fit to what is trying to be done (map JSON named parameters to the correct method).

Another option entirely is like what many others do... a sort of "IDL" that generates .cpp as a compile step (i.e. JsonRpcCpp, CORBA, Protobuf, et.al.).

Feel free to shoot holes in this...

EDIT: I have put together a POC, and while the article on the website above does help with named parameters and defaults, It does not help with mapping parameters as string (i.e. coming from JSON) to their class counterpart. Stringifying and building a map at compile time is the only way I can see to do that. But for all that, why not do option 2 (IDL)?

[abitmore]

Hi @jmjatlanta, thanks for the comments. Personally I don't want to touch this ticket since I'm not a fan of deep code refactoring. Wish someone else can help you and/or work together with you. Wish you good luck.

[jmjatlanta]

Thanks for the reply @abitmore. Even the "IDL" idea has some drawbacks. 1) the extra compile step, 2) performance.

The way it is done now is with variadic templates. This requires parameters in a specific order, and no defaults unless you want make a specialized template (which kind of defeats the purpose of the template). The pros are that it is fast, with some compile-time type checking.

The way I see the implementation of @vikramrajkumar 's comment above would require a map of field names with fields, which would certainly not be as performant. I guess the decision must be made. Do we want a maintainable, defaultable, and expandable API that is not as fast, or stay with the status quo that works, is fast, but is missing some features.

Just a note here in case I (or someone else) don't get back here for a while: We can decipher the field number and its type if we want to, as part of @vikramrajkumar 's suggestion. The problem is still mapping that parameter to its name, so that we can give it a value. IMO if we're going to build maps of parameter names, we should also look at something like JsonRpcCpp.