steemd-compatible APIs #19

roadscape · 2017-09-25T20:17:25Z

Responses should be as close as possible to steemd, making it easy for devs to switch to hive APIs.

roadscape · 2017-09-28T19:48:14Z

The full set of fields from steemd:

"roadscape/hello-world": {
  "id": 73,
  "author": "roadscape",
  "permlink": "hello-world",
  "category": "random",
  "parent_author": "",
  "parent_permlink": "random",
  "title": "Hello World!",
  "body": "Hello World ⚡ - Здравствулте мир - 你好世界 - Bonjour monde - Hallo Welt - γειά σου κόσμος - こんにちは世界 - 여보세요 세계",
  "json_metadata": "{}",
  "last_update": "2016-04-11T02:27:48",
  "created": "2016-04-11T02:27:48",
  "active": "2017-06-29T22:52:57",
  "last_payout": "2016-08-20T17:03:00",
  "depth": 0,
  "children": 3,
  "net_rshares": 0,
  "abs_rshares": 0,
  "vote_rshares": 0,
  "children_abs_rshares": 0,
  "cashout_time": "1969-12-31T23:59:59",
  "max_cashout_time": "1969-12-31T23:59:59",
  "total_vote_weight": 0,
  "reward_weight": 10000,
  "total_payout_value": "0.480 SBD",
  "curator_payout_value": "0.480 SBD",
  "author_rewards": 2188,
  "net_votes": 7,
  "root_comment": 73,
  "max_accepted_payout": "1000000.000 SBD",
  "percent_steem_dollars": 10000,
  "allow_replies": true,
  "allow_votes": true,
  "allow_curation_rewards": true,
  "beneficiaries": [],
  "url": "/random/@roadscape/hello-world",
  "root_title": "Hello World!",
  "pending_payout_value": "0.000 SBD",
  "total_pending_payout_value": "0.000 STEEM",
  "active_votes": [{
      "voter": "puppies",
      "weight": "1811736000000",
      "rshares": 1811736,
      "percent": 10000,
      "reputation": 0,
      "time": "2016-04-11T05:40:57"
    },
    ...
  ],
  "replies": [
    "stellabelle/re-roadscape-hello-world-20160612t005210219z",
    "gerka23/re-roadscape-hello-world-20160720t210643160z",
    "troilo/re-roadscape-hello-world-20170629t225257985z"
  ],
  "author_reputation": 0,
  "promoted": "0.000 SBD",
  "body_length": 0,
  "reblogged_by": []
},

Condenser also makes heavy use of these calculated 'stats':

{
  hide, 
  gray, 
  authorRepLog10, 
  allowDelete, 
  isNsfw, 
  flagWeight, 
  total_votes, 
  up_votes, 
  hasPendingPayout
}

The steemd response is quite verbose, so the goal is to get rid of any rarely-used fields, keep widely-used fields, and replace a few with derived data which would be more convenient anyway.

roadscape · 2017-12-04T20:50:42Z

Based on condenser:

Not used

  "id": 73,
  "root_comment": 73,

  "active": "2017-06-29T22:52:57",
  "last_payout": "2016-08-20T17:03:00",
  "max_cashout_time": "1969-12-31T23:59:59",

  "abs_rshares": 0,
  "vote_rshares": 0,
  "children_abs_rshares": 0,

  "net_votes": 7,
  "total_vote_weight": 0,
  "reward_weight": 10000,
  "author_rewards": 2188,

  "total_pending_payout_value": "0.000 STEEM", -- deprecated?

Not used - potentially important

  "allow_replies": true,
  "allow_votes": true,
  "allow_curation_rewards": true,
  "beneficiaries": [],

Important - core fields

  "author": "roadscape",
  "permlink": "hello-world",
  "category": "random",
  "created": "2016-04-11T02:27:48",
  "last_update": "2016-04-11T02:27:48",

  "parent_author": "",
  "parent_permlink": "random",

  "title": "Hello World!",
  "body": "Hello World ⚡ - Здравствулте мир - 你好世界 - Bonjour monde - Hallo Welt - γειά σου κόσμος - こんにちは世界 - 여보세요 세계",
  "json_metadata": "{}",

  "net_rshares": 0,
  "cashout_time": "1969-12-31T23:59:59",
  "total_payout_value": "0.480 SBD",
  "curator_payout_value": "0.480 SBD",
  "pending_payout_value": "0.000 SBD",
  "promoted": "0.000 SBD",

Important - derived

  "depth": 0,
  "children": 3,
  "replies": [
    "stellabelle/re-roadscape-hello-world-20160612t005210219z",
    "gerka23/re-roadscape-hello-world-20160720t210643160z",
    "troilo/re-roadscape-hello-world-20170629t225257985z"
  ],

In use - to replace/remove?

  "author_reputation": 0,                    -- move to author object?
  "reblogged_by": []                         -- this will be added dynamically to get_discussions APIs
  "max_accepted_payout": "1000000.000 SBD",  -- only used to determine if payout_declined
  "percent_steem_dollars": 10000,            -- only used to determine if 100% powered up
  "body_length": 0,                          -- only used to determine if body was truncated in response
  "url": "/random/@roadscape/hello-world",   -- only used in PostSummary, PostFull
  "root_title": "Hello World!",  -- only used in PostSummary, PostFull
  "active_votes": [{                         -- can be replaced with more lightweight "top 20"
      "voter": "puppies",                      -- keep
      "weight": "1811736000000",               -- not used
      "rshares": 1811736,                      -- keep (useful for sorting)
      "percent": 10000,                        -- keep
      "reputation": 0,                         -- not needed if we pre-process
      "time": "2016-04-11T05:40:57"            -- not needed?
    },
    ...
  ],

Condenser derived fields

  hide,             -- calculate only on retrieval?
  gray,             -- calculate only on retrieval?
  authorRepLog10,   -- move to author object?
  allowDelete,      -- could replace with `post['children'] == 0 and int(post['net_rshares']) <= 0`
  isNsfw,           -- keep
  flagWeight,       -- keep
  total_votes,      -- keep
  up_votes,         -- not used
  hasPendingPayout  -- could easily replace w/ `post['pending_payout_value'] >= 0.02`

bonustrack · 2017-12-05T11:03:02Z

For busy.org:

Here are the fields we are using currently:
[Used]
"id": 73,
"net_votes": 7,

We plan to use this ones (could be hidden if using default value):
"allow_replies": true,
"allow_votes": true,
"allow_curation_rewards": true,
"beneficiaries": [],

We also use these ones but can move if necessary:
"author_reputation": 0,
"reblogged_by": [] // We plan to use it
"max_accepted_payout": "1000000.000 SBD",
"percent_steem_dollars": 10000,
"url": "/random/@roadscape/hello-world",
"root_title": "Hello World!",
"active_votes": [{ // Used with full list actually
"voter": "puppies",
"rshares": 1811736,
"percent": 10000,
"reputation": 0, // Not used but plan to
"time": "2016-04-11T05:40:57",
}],

feruzm · 2017-12-05T11:27:57Z

Based on eSteem:

Not used

  "last_payout": "2016-08-20T17:03:00",
  "net_votes": 7,
  "total_pending_payout_value": "0.000 STEEM"

ONLY ABOVE IS USED

Not used - potentially important

NOT USED YET

Important - core fields

USED

In use - to replace/remove?

USED

roadscape · 2017-12-08T19:54:11Z

@bonustrack, @feruzm thanks for the input! Here's some more questions:

"id": 73,

id is an internal field which (IIRC) is not guaranteed to be the same between witness_nodes. What do you use it for? What if I served the hive internal post id instead?

"total_payout_value": "0.480 SBD",
"curator_payout_value": "0.480 SBD",
"pending_payout_value": "0.000 SBD",

Currently, hive sums these 3 fields. It's just payout (decimal amount) and is_paidout (bool). Is curator payout value important? I can add a compatibility layer to set these 3 fields but it would be easier if I can ignore curator split (I would set it to 0 and add it to the other field).

"max_accepted_payout": "1000000.000 SBD", -- only used to determine if payout_declined
"percent_steem_dollars": 10000, -- only used to determine if 100% powered up

What do you guys use these for? Are you interested in raw amount or just a boolean "payout_declined"/"fully_powered_up"?

"active_votes": [{ // Used with full list actually
"voter": "puppies",
"rshares": 1811736,
"percent": 10000,
"reputation": 0, // Not used but plan to
"time": "2016-04-11T05:40:57",
}],

active_votes is by far the heaviest part of content responses, so it's important to slim it down somehow. For instance, do we really need reputation and time for each vote? Also, for a post with e.g. 3000 wouldn't loading the top 20-50 votes be sufficient?

mvandeberg · 2017-12-08T20:18:04Z

id is an internal field which (IIRC) is not guaranteed to be the same between witness_nodes. What do you use it for? What if I served the hive internal post id instead?

id should be the same across nodes (if it is not, please report it). However, we treat it as an implementation detail of our database. Meaning, if Steem we implemented on top of an SQL database, the id field might not be needed.

I would transition busy.org away from using id unless there is a compelling reason not to.

feruzm · 2017-12-08T20:41:34Z

What do you guys use these for? Are you interested in raw amount or just a boolean "payout_declined"/"fully_powered_up"?

Well, I imagine some UIs might offer different set of reward sharing and removing those kinda feels limiting some possibilities but if it is overhead we might as well not include that or change as suggested

active_votes is by far the heaviest part of content responses, so it's important to slim it down somehow. For instance, do we really need reputation and time for each vote? Also, for a post with e.g. 3000 wouldn't loading the top 20-50 votes be sufficient?

active_votes is not included in all blockchain APIs so eSteem using get_active_votes api separately to show voters info (voter, reputation, time, percent fields are used)

roadscape · 2017-12-08T20:52:03Z

@feruzm: with regards to active_votes, how do you make use reputation/time values?

Do you need all votes or would top 50 with extra stats (total votes, net votes, vote state of current user) be enough?

feruzm · 2017-12-08T20:58:44Z

When user taps on number of votes additional screen gives them some details:

…

On 8 Dec 2017, at 20:52, roadscape ***@***.***> wrote: @feruzm: with regards to active_votes, how do you make use reputation/time values? Do you need all votes or would top 50 with extra stats (total votes, net votes, vote state of current user) be enough? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

feruzm · 2017-12-08T21:03:54Z

roadscape · 2017-12-08T22:35:44Z

@feruzm is that option available for "post lists" or only on viewing specific post?

bonustrack · 2017-12-09T05:13:10Z

"id": 73,

About id we use it to store post in redux state. Actually we can deal with it if you remove it or change with another id.

"total_payout_value": "0.480 SBD",
"curator_payout_value": "0.480 SBD",
"pending_payout_value": "0.000 SBD",

We use these value to generate this tooltip:

I'm not sure if its possible to calculate these payout without the values, the curator split is never the same % and I feel its important value to have for end user. Any idea?

"max_accepted_payout": "1000000.000 SBD", -- only used to determine if payout_declined
"percent_steem_dollars": 10000, -- only used to determine if 100% powered up
If we use "payout_declined"/"fully_powered_up" it would be fine for actual busy usecase, but in future we might want to show the percent of steem dollar.

For active_votes we use it here:

And here:

On the tooltip we show the 10 biggest votes, then there is the full list when you click on the vote counter. I understand its not efficient at all to add the full list of upvote on the feed response, we probably can switch to use get_active_votes to show the full list, but we will need to have at least these 10 most powerfull vote (vote/downvote included). The reputation and time is not necessary for busy.org to generate the tooltip but we want it for do a full list if we use get_active_votes.

feruzm · 2017-12-09T08:38:41Z

@roadscape it is on all pages, post, list, profile blog, etc.

roadscape · 2017-12-11T16:48:02Z

It seems the best way forward is to create a compatibility layer which will be mostly a drop-in replacement, to give devs more time to switch to the new & improved API.

#41 is relevant to backwards-compatible pagination. If this will be available in 1.0 or shortly after is TBD. It should probably be considered deprecated from the start.

roadscape · 2017-12-14T01:06:48Z

Ok, for 1.0 it's now a requirement to support the legacy API request/response formats.

Here are the ones I've identified:

Discussions Queries

get_discussions_by_trending           {tag, limit, start_author, start_permlink}    trending
get_discussions_by_hot                {tag, limit, start_author, start_permlink}    hot
get_discussions_by_promoted           {tag, limit, start_author, start_permlink}    promoted
get_discussions_by_created            {tag, limit, start_author, start_permlink}    created
get_discussions_by_blog               {tag, limit, start_author, start_permlink}    account-blog
get_discussions_by_feed               {tag, limit, start_author, start_permlink}    account-feed
get_discussions_by_comments                {limit, start_author, start_permlink}    account-comments
get_replies_by_last_update            [author, permlink, limit];                    account-replies

Deprecated Discussions Queries (no plans to implement in hive)

get_post_discussions_by_payout        {tag, limit, start_author, start_permlink}    payout
get_comment_discussions_by_payout     {tag, limit, start_author, start_permlink}    payout_comments
get_discussions_by_cashout            {tag, limit, start_author, start_permlink}    UNUSED/cashout
get_discussions_by_children           {tag, limit, start_author, start_permlink}    UNUSED/responses
get_discussions_by_votes              {tag, limit, start_author, start_permlink}    UNUSED/votes
get_discussions_by_active             {tag, limit, start_author, start_permlink}    DEPRECATED
get_discussions_by_trending30         {tag, limit, start_author, start_permlink}    DEPRECATED
get_discussions_by_payout             {tag, limit, start_author, start_permlink}    DEPR/UNUSED
get_discussions_by_author_before_date [author, start_permlink, before_date, limit]  DEPR/UNUSED

Follows Queries

get_followers                         [following, start_follower, follow_type, limit]
get_following                         [follower, start_following, follow_type, limit]
get_follow_count                      [account];

Posts Queries

get_content                           [author, permlink]
get_content_replies                   [parent_author, parent_permlink]

There's an issue with get_state; it includes a lot of data that is out of hivemind's scope. I will evaluate what it would take to replicate it, otherwise hive will need to forward the request to steemd and supplement/modify the response as necessary.

roadscape · 2017-12-14T01:32:25Z

I've made an issue in the steemd repo regarding removal of the 6 deprecated queries: steemit/steem#1903 -- please comment there is any are still in use.

roadscape · 2017-12-14T19:11:33Z

Request formats and functionality now implemented for:

vector< follow_api_obj > get_followers( string following, string start_follower, follow_type type, uint16_t limit )const;
vector< follow_api_obj > get_following( string follower, string start_following, follow_type type, uint16_t limit )const;
follow_count_api_obj     get_follow_count( string& account )const;

vector<discussion> get_discussions_by_trending( const discussion_query& query )const;
vector<discussion> get_discussions_by_created( const discussion_query& query )const;
vector<discussion> get_discussions_by_hot( const discussion_query& query )const;
vector<discussion> get_discussions_by_feed( const discussion_query& query )const;
vector<discussion> get_discussions_by_blog( const discussion_query& query )const;
vector<discussion> get_discussions_by_comments( const discussion_query& query )const;
vector<discussion> get_discussions_by_promoted( const discussion_query& query )const;

Need to fix non-standard request format (takes array, not dict):

vector<discussion> get_replies_by_last_update( account_name_type start_author, string start_permlink, uint32_t limit )const;

TODO: For the above calls, implement steemd-formatted responses.

To implement:

state                get_state( string path )const;
discussion           get_content( string author, string permlink )const;
vector<discussion>   get_content_replies( string parent, string parent_permlink )const;

Not used:

vector<discussion> get_discussions_by_payout(const discussion_query& query )const;
vector<discussion> get_post_discussions_by_payout( const discussion_query& query )const;
vector<discussion> get_comment_discussions_by_payout( const discussion_query& query )const;
vector<discussion> get_discussions_by_active( const discussion_query& query )const;
vector<discussion> get_discussions_by_cashout( const discussion_query& query )const;
vector<discussion> get_discussions_by_votes( const discussion_query& query )const;
vector<discussion> get_discussions_by_children( const discussion_query& query )const;
vector<discussion> get_discussions_by_author_before_date( string author, string start_permlink, time_point_sec before_date, uint32_t limit )const;

roadscape · 2017-12-21T23:37:01Z

The following have been implemented in hive and tested to work with condenser. get_state supports a minimal subset of steemd features which condenser relies on.

    condenser_api.get_followers,
    condenser_api.get_following,
    condenser_api.get_follow_count,

    condenser_api.get_discussions_by_trending,
    condenser_api.get_discussions_by_hot,
    condenser_api.get_discussions_by_promoted,
    condenser_api.get_discussions_by_created,
    condenser_api.get_discussions_by_blog,
    condenser_api.get_discussions_by_feed,
    condenser_api.get_discussions_by_comments,
    condenser_api.get_replies_by_last_update,

    condenser_api.get_content,
    condenser_api.get_content_replies,

    condenser_api.get_state

Closing; any minor issues w/ API can be filed under "frontend implementation" #33

roadscape added the priority/high label Sep 25, 2017

roadscape changed the title ~~maximize compatibility between hive and steemd discussion APIs~~ discussion APIs should resemble steemd's Nov 17, 2017

roadscape changed the title ~~discussion APIs should resemble steemd's~~ discussion APIs should match steemd Nov 17, 2017

roadscape added this to the 1.0 milestone Nov 17, 2017

roadscape added priority/med and removed priority/high labels Nov 22, 2017

roadscape added the WIP label Dec 1, 2017

roadscape added priority/high and removed priority/med labels Dec 13, 2017

roadscape changed the title ~~discussion APIs should match steemd~~ steemd-compatible APIs Dec 14, 2017

roadscape mentioned this issue Dec 14, 2017

thread fetching API #29

Closed

This was referenced Dec 14, 2017

legacy api #45

Merged

cursor-style pagination? #41

Closed

roadscape self-assigned this Dec 21, 2017

roadscape closed this as completed Dec 21, 2017

roadscape removed the WIP label Dec 21, 2017

roadscape mentioned this issue Feb 17, 2018

update readme / docs #49

Closed

10 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

steemd-compatible APIs #19

steemd-compatible APIs #19

roadscape commented Sep 25, 2017

roadscape commented Sep 28, 2017

roadscape commented Dec 4, 2017

bonustrack commented Dec 5, 2017

feruzm commented Dec 5, 2017

Not used

Not used - potentially important

Important - core fields

In use - to replace/remove?

roadscape commented Dec 8, 2017

mvandeberg commented Dec 8, 2017

feruzm commented Dec 8, 2017

roadscape commented Dec 8, 2017

feruzm commented Dec 8, 2017 via email

feruzm commented Dec 8, 2017

roadscape commented Dec 8, 2017

bonustrack commented Dec 9, 2017

feruzm commented Dec 9, 2017

roadscape commented Dec 11, 2017

roadscape commented Dec 14, 2017 •

edited

Loading

roadscape commented Dec 14, 2017

roadscape commented Dec 14, 2017

roadscape commented Dec 21, 2017

steemd-compatible APIs #19

steemd-compatible APIs #19

Comments

roadscape commented Sep 25, 2017

roadscape commented Sep 28, 2017

roadscape commented Dec 4, 2017

Not used

Not used - potentially important

Important - core fields

Important - derived

In use - to replace/remove?

Condenser derived fields

bonustrack commented Dec 5, 2017

feruzm commented Dec 5, 2017

Not used

Not used - potentially important

Important - core fields

In use - to replace/remove?

roadscape commented Dec 8, 2017

mvandeberg commented Dec 8, 2017

feruzm commented Dec 8, 2017

roadscape commented Dec 8, 2017

feruzm commented Dec 8, 2017 via email

feruzm commented Dec 8, 2017

roadscape commented Dec 8, 2017

bonustrack commented Dec 9, 2017

feruzm commented Dec 9, 2017

roadscape commented Dec 11, 2017

roadscape commented Dec 14, 2017 • edited Loading

roadscape commented Dec 14, 2017

roadscape commented Dec 14, 2017

roadscape commented Dec 21, 2017

roadscape commented Dec 14, 2017 •

edited

Loading