Fix #1270 Call price is inconsistent when MCR changed #1324

abitmore · 2018-09-13T21:39:00Z

PR for #1270.

test cases
- modify old test cases related to margin calls, see if anything is broken after the new hard fork time
- reproduce the issue before the hard fork time, aka changing MCR doesn't trigger margin calls (or push some call orders out of margin call territory)
  - go over the hard fork time, check if margin calls are triggered (or no longer triggered) as expected
- changing MCR again after the hard fork time, check if margin calls are triggered correctly

Note: this should not break compatibility in production because the result should be the same after hard fork #343.

oxarbitrage · 2018-11-13T19:18:00Z

@abitmore, let me know what do you think about this possible framework to modify current testcases for the new after HF logic, i am trying to do this without duplicating too much code.

added flag to the top: https://github.com/oxarbitrage/bitshares-core/blob/mcr_bug_testscases/tests/tests/market_tests.cpp#L36
flag is checked on top of each case: https://github.com/oxarbitrage/bitshares-core/blob/mcr_bug_testscases/tests/tests/market_tests.cpp#L45-L52
flag is modified and test called with new hf date: https://github.com/oxarbitrage/bitshares-core/blob/mcr_bug_testscases/tests/tests/market_tests.cpp#L1582-L1589

With the flag available inside each test case we can change the logic of them to match/test the new features for the existing test cases.

abitmore · 2018-11-18T18:41:07Z

@oxarbitrage we can have a better test framework for hard forks, but I don't think your solution is good enough. It's incorrect to add a flag to all test cases, instead, we should do it on a case by case basis so far. It doesn't make much sense to have one flag on the top, and worse, we (e.g. Travis) won't modify it when running test cases.

Ideally some test cases should be executed once after EVERY hard fork time, from this point of view, we should modify all the test cases, and have a mechanism to avoid future modifications. IMHO this is out of this PR's scope, although it may be helpful for future work (need to make a decision about priority).

Update:

if need a flag which can be used in different test cases, put it in database_fixture. Using a global variable is troublesome.
your new test cases are failing, that (imho) means they are not well designed. Need to understand them and modify only necessary parts.

oxarbitrage · 2018-11-18T19:25:17Z

your new test cases are failing, that (imho) means they are not well designed. Need to understand them and modify only necessary parts.

Yes, that is what i am trying but i was thinking on modifying only the needed parts with the flag in the same test case.

My approach was/is to identify where they fail and then try to understand why and workaround. I was able to identify where they fail in the first 2,3 cases with this.

Anyways, i understand your point, can create separated new test cases for each of the current tests advancing to HF date and modify what is needed there, at least by now.

Thanks.

oxarbitrage · 2018-12-12T02:36:08Z

@abitmore please take a look when you can at: oxarbitrage@d9bb527

specifically, the cross test(last one). thank you.

add support for existing tests after hf1270

Resolved conflicts: - libraries/chain/db_maint.cpp - libraries/chain/include/graphene/chain/config.hpp - libraries/chain/include/graphene/chain/database.hpp - tests/tests/swan_tests.cpp

pmconrad

Very good. Only "serious" issue is virtual op IDs in global settlement.

Is it possible to get rid of a sub-index in a boost multi_index? That would free some resources. Would have to be delayed until LIB is past the HF time though.

libraries/chain/asset_evaluator.cpp

libraries/chain/market_object.cpp

pmconrad · 2019-01-31T16:22:09Z

libraries/chain/db_maint.cpp

+   wlog( "Done updating all call orders for hardfork core-343 at block ${n}", ("n",db.head_block_num()) );
+}
+
+/// Reset call_price of all call orders to (1,1) since it won't be used in the future.


This is superfluous, since as you say it won't used anymore in the future. Getting rid of this and not tying the hf to a maintenance interval will simplify the PR significantly.

My concerns:

For UI. If old positions' call_price are unchanged but new positions' are 1, people may get confused.

I guess it's faster to compare 1 with 1 when inserting new data (when creating new positions) so it benefits in the long run with the cost of one-time process.

BTW as you've noticed later we need to tie the hf to a maintenance interval anyway.

pmconrad · 2019-01-31T16:31:07Z

libraries/chain/db_market.cpp

@@ -62,15 +62,14 @@ void database::globally_settle_asset( const asset_object& mia, const price& sett
   const asset_dynamic_data_object& mia_dyn = mia.dynamic_asset_data_id(*this);
   auto original_mia_supply = mia_dyn.current_supply;

-   const call_order_index& call_index = get_index_type<call_order_index>();
-   const auto& call_price_index = call_index.indices().get<by_price>();
+   const auto& call_index = get_index_type<call_order_index>().indices().get<by_collateral>();


Switching to a different index here has the potential to change the order in which call orders are closed.
This shouldn't change the outcome in terms of amounts, but it can change the ID of virtual operations. We usually try to avoid that.

Will revert this.

Done via b494609.

pmconrad · 2019-01-31T16:43:36Z

libraries/chain/asset_object.cpp

@@ -100,6 +100,9 @@ void graphene::chain::asset_bitasset_data_object::update_median_feeds(time_point
   if( current_feed.core_exchange_rate != median_feed.core_exchange_rate )
      feed_cer_updated = true;
   current_feed = median_feed;
+   // Note: perhaps can defer updating current_maintenance_collateralization for better performance
+   if( after_core_hardfork_1270 )
+      current_maintenance_collateralization = current_feed.maintenance_collateralization();


Unconditionally, i. e. always, setting this will simplify the code significantly. I believe the performance penalty will be insignificant since you get rid of a method parameter and several checks at the same time.

Because there are several 128-bit multiplications and divisions in maintenance_collateralization(), I thought that it costs more. Anyway we can benchmark it (I haven't done).

libraries/chain/db_maint.cpp

Because all positions will be closed anyway, using which index doesn't matter in terms of amounts, however, using another index may change the ID of historical virtual operations, specifically, `fill_order_operation` -- we usually try to avoid this.

abitmore · 2019-02-01T11:21:28Z

WRT getting rid of a sub-index in a boost multi_index, I have an idea:

replace it with a secondary index, which has a variable indicates whether the secondary index is enabled;
- before the hard fork, we use the secondary index;
- at hf time (or after LIB passed hf time?) we update the variable, so the secondary index will be disabled thus will save resources in the future.

I guess it's possible to be implemented, however it would be a bit complicated, I'm not sure if it worth the efforts.

pmconrad · 2019-02-04T14:28:11Z

Good to go, IMO. Any potential optimizations can be implemented after the release.

pmconrad

Thanks!

abitmore added 15 commits September 13, 2018 11:59

BitAsset obj: added maintenance_collateralization

84c7218

Refactored feed_price::max_short_squeeze_price()

6254076

Updated comment in price::call_price()

9b41a92

Iterates by collateral in globally_settle_asset()

5308ea0

Iterate call orders by collateral in apply_order()

c912886

Iterate calls by collateral in check_call_orders()

469a499

db_market: stop using call_price after hf #1270

cb7bd0c

Do not update current_maint_collateral before hf

0ba6cb1

check revive: stop using call_price after hf #1270

92a3b71

update call: stop using call_price after hf #1270

077f4e0

Update and match call orders at hf #1270

68c702b

Check for blackswan by collateral after hf #1270

8bcf38d

get_call_orders() database API: use by_collateral

fd1c091

Note: this should not break compatibility in production because the result should be the same after hard fork #343.

Minor coding style update

129e89c

Bump DB_VERSION due to call_order_object change

9af57a8

abitmore added the hardfork label Sep 13, 2018

abitmore added this to the 201903 - Consensus-Changing Release milestone Sep 13, 2018

abitmore self-assigned this Sep 13, 2018

abitmore added this to In Development in Protocol Upgrade Release (3.0.0) Sep 13, 2018

Target CR: stop using call_price after hf #1270

b5e63a3

ryanRfox mentioned this pull request Nov 8, 2018

Test Cases for PR 1270 (MCR Bugfix) #1423

Closed

10 tasks

oxarbitrage self-requested a review November 9, 2018 17:57

abitmore modified the milestones: 201811 - Consensus-Upgrade Release, 201905 - Consensus-Upgrade Release Nov 12, 2018

oxarbitrage added 2 commits December 15, 2018 17:47

update median feeds when crossing hf1270

a56562a

add testcases for mcr bug(hf1270)

d7306ab

oxarbitrage and others added 10 commits December 27, 2018 13:19

add hf1270 swan tests

46e2943

skip black_swan_issue_346 test after hf1270

4e2514c

update bitasset tests for hf1270

7abf7c5

accurate block generation for hf1270 in market and swan tests

a2d5fac

changes in bitasset_tests/hf_935_test

8466e19

workaround hf_935_test

ece99b6

use affected_by_hf_343 flag

1aad691

remove trailing white spaces

ca1e045

Merge pull request #1493 from oxarbitrage/1270_existing_testcases

6b22b91

add support for existing tests after hf1270

Merge hardfork branch

54c4c02

Resolved conflicts: - libraries/chain/db_maint.cpp - libraries/chain/include/graphene/chain/config.hpp - libraries/chain/include/graphene/chain/database.hpp - tests/tests/swan_tests.cpp

abitmore requested a review from pmconrad January 30, 2019 19:27

pmconrad suggested changes Jan 31, 2019

View reviewed changes

Protocol Upgrade Release (3.0.0) automation moved this from In Development to In Testing Feb 4, 2019

pmconrad previously approved these changes Feb 4, 2019

View reviewed changes

Simplify lambda

a22f92e

abitmore dismissed pmconrad’s stale review via a22f92e February 4, 2019 20:33

Protocol Upgrade Release (3.0.0) automation moved this from In Testing to In Development Feb 4, 2019

abitmore requested review from pmconrad and jmjatlanta February 4, 2019 20:35

pmconrad moved this from In Development to In Review in Protocol Upgrade Release (3.0.0) Feb 5, 2019

pmconrad approved these changes Feb 5, 2019

View reviewed changes

abitmore merged commit aff1751 into hardfork Feb 5, 2019

Protocol Upgrade Release (3.0.0) automation moved this from In Review to Done Feb 5, 2019

abitmore deleted the 1270-fix-call-price-cache branch February 5, 2019 20:42

abitmore mentioned this pull request Feb 5, 2019

Call price is inconsistent when MCR changed #1270

Closed

17 tasks

MichelSantos mentioned this pull request Feb 8, 2019

Release Notes: BitShares Core version 3.0.0 #1569

Closed

abitmore mentioned this pull request Mar 23, 2019

Stop using call_price when globally settling after the hard fork #1669

Closed

17 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix #1270 Call price is inconsistent when MCR changed #1324

Fix #1270 Call price is inconsistent when MCR changed #1324

abitmore commented Sep 13, 2018 •

edited

oxarbitrage commented Nov 13, 2018 •

edited

abitmore commented Nov 18, 2018 •

edited

oxarbitrage commented Nov 18, 2018

oxarbitrage commented Dec 12, 2018

pmconrad left a comment

pmconrad Jan 31, 2019

abitmore Feb 1, 2019

pmconrad Jan 31, 2019

abitmore Feb 1, 2019

abitmore Feb 1, 2019

pmconrad Jan 31, 2019

abitmore Feb 1, 2019

abitmore commented Feb 1, 2019

pmconrad commented Feb 4, 2019

pmconrad left a comment

Fix #1270 Call price is inconsistent when MCR changed #1324

Fix #1270 Call price is inconsistent when MCR changed #1324

Conversation

abitmore commented Sep 13, 2018 • edited

oxarbitrage commented Nov 13, 2018 • edited

abitmore commented Nov 18, 2018 • edited

oxarbitrage commented Nov 18, 2018

oxarbitrage commented Dec 12, 2018

pmconrad left a comment

Choose a reason for hiding this comment

pmconrad Jan 31, 2019

Choose a reason for hiding this comment

abitmore Feb 1, 2019

Choose a reason for hiding this comment

pmconrad Jan 31, 2019

Choose a reason for hiding this comment

abitmore Feb 1, 2019

Choose a reason for hiding this comment

abitmore Feb 1, 2019

Choose a reason for hiding this comment

pmconrad Jan 31, 2019

Choose a reason for hiding this comment

abitmore Feb 1, 2019

Choose a reason for hiding this comment

abitmore commented Feb 1, 2019

pmconrad commented Feb 4, 2019

pmconrad left a comment

Choose a reason for hiding this comment

abitmore commented Sep 13, 2018 •

edited

oxarbitrage commented Nov 13, 2018 •

edited

abitmore commented Nov 18, 2018 •

edited