Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent excessive api calls #4966

Merged
merged 54 commits into from Dec 23, 2020
Merged

Commits on Dec 4, 2020

  1. Refactor BtcWalletService to let api override fee rates

    BtcWalletService was changed to allow the api to override tx fee
    rates from the sendbsq and sendbtc methods.  The api methods will
    still be able to use the network fee service and custom tx fee rate
    preference, and set / unset the custom tx fee rate preference, but
    the change will permit the addition of an optional txFeeRate parameter
    to the sendbsq and sendbtc methods (todo).  A few other minor changes
    (style and removal of never thrown ex spec) were also made to this class.
    
    Two BtcWalletService methods were refactored.
    
    - The redundant (was always true) boolean isSendTx argument was removed
      from the completePreparedVoteRevealTx method signature.
    
    - The redundant (was always true) boolean useCustomTxFee was removed
      from the completePreparedBsqTx method signature.
    
    - The completePreparedSendBsqTx method was overloaded with a 2nd parameter
      (Coin txFeePerVbyte) to allow api to override fee service and custom
      tx fee rate when sending BSQ or BTC.
    
    - The completePreparedBsqTx method was overloaded with a 3rd parameter
      (Coin txFeePerVbyte) to allow api to override fee service and custom
      tx fee rate when sending BSQ or BTC.
    
    The following line was deleted from the completePreparedBsqTx method
    because txFeePerVbyte is now an argument:
    
    	Coin txFeePerVbyte = useCustomTxFee ? getTxFeeForWithdrawalPerVbyte() : feeService.getTxFeePerVbyte();
    
    This useCustomTxFee value was always true, and redudant here because
    getTxFeeForWithdrawalPerVbyte() returns feeService.getTxFeePerVbyte()
    or the custom fee rate preference. i.e.,
    
    Coin txFeePerVbyte = useCustomTxFee ? getTxFeeForWithdrawalPerVbyte() : feeService.getTxFeePerVbyte();
    
    	is equivalent to
    
    Coin txFeePerVbyte = getTxFeeForWithdrawalPerVbyte();
    
    LockupTxService, UnlockTxService, BsqSendView, and BsqTransferService
    were adjusted to this BtcWalletService refactoring.
    ghubstan committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    dc6144d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    900d498 View commit details
    Browse the repository at this point in the history
  3. Add optional txFeeRate parameter to api sendbsq

    If present in the sendbsq command, the parameter will override the fee
    service and custom fee rate setting for the BSQ transaction.
    
    Also changed the sendbsq grpc return type to a lightweight TX proto wrapper.
    
    Besides some small refactoring in the CLI, all the changes are
    adjustments for this new sendbsq parameter and its new grpc return value.
    ghubstan committed Dec 4, 2020
    Configuration menu
    Copy the full SHA
    159d4cc View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2020

  1. Configuration menu
    Copy the full SHA
    2842070 View commit details
    Browse the repository at this point in the history

Commits on Dec 9, 2020

  1. Add new api method 'sendbtc' and test

    Takes an address, amount, and optional txfeerate param,
    returns a lightweight TxInfo proto.
    
    Also overloaded two BtcWalletService methods to allow sendbtc
    to pass in the tx fee rate -- overriding the fee service and
    custom fee rate setting.
    ghubstan committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    6c9f0c2 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    144c5a8 View commit details
    Browse the repository at this point in the history
  3. Support tx memo field for btc withdrawals from api

    - Added optional memo parameter to the api's sendbtc and
      withdrawfunds commands.
    
    - Removed the @nullable annotation was removed because protobuf
      does not support null.
    
    - Visibility in two wallet check methods were changed from private
      to pkg protected so the CoreTradeService could use them.
    
    - Adjusted affected tests.  (Asserting the memo field was set on a
      transaction cannot be checked from apitest yet.)
    ghubstan committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    bd66008 View commit details
    Browse the repository at this point in the history
  4. Remove unused imports

    ghubstan committed Dec 9, 2020
    Configuration menu
    Copy the full SHA
    478c8f4 View commit details
    Browse the repository at this point in the history

Commits on Dec 10, 2020

  1. Configuration menu
    Copy the full SHA
    259bad6 View commit details
    Browse the repository at this point in the history

Commits on Dec 11, 2020

  1. Configuration menu
    Copy the full SHA
    150e2f6 View commit details
    Browse the repository at this point in the history

Commits on Dec 12, 2020

  1. Append nullable withdrawalTxId field to Trade proto message

    The withdrawalTxId field will be set in TradeManager#onWithdrawRequest
    upon successful trade completion and withdrawal of funds.
    
    Persisting withdrawalTxId will allow the api and ui to find the withdrawalTxId
    for a completed trade after the seller withdraws funds to an external wallet.
    In turn, the withdrawal tx's memo field will be accessible in a new (todo)
    api getTx(txID) api method.
    
    Changed:
    
    - Appended field 'string withdrawal_tx_id = 40' to pb.proto's Trade message.
    
    - Added nullable 'String withdrawalTxId' to Trade entity class.
    
    - Added trade.setWithdrawalTxId(transaction.getTxId().toString()) in
      TradeManager#onWithdrawRequest's callback.
    ghubstan committed Dec 12, 2020
    Configuration menu
    Copy the full SHA
    6aa385e View commit details
    Browse the repository at this point in the history

Commits on Dec 14, 2020

  1. Add new api method gettransaction

    This change was prompted by the recent changes in the main branch to
    allow a tx memo field to be set from the UI and API.
    
    This and the prior PR address the API's need to be able to fetch a
    tx (with a memo).  The API can now get a completed trade's withdrawal
    txid and pass it as a gettransaction parameter.
    
    See previous PR "Append nullable withdrawalTxId field to Trade".
    
    	bisq-network#4937
    
    A summary of changes by file:
    
    grpc.proto
    
    - Added withdrawalTxId field to existing TradeInfo proto & wrapper.
    - Reordered fields in TradeInfo proto.
    - Added new fields to be displayed by TxInfo proto in CLI.
    - Fixed typo: unsetTxFeeRatePreference -> UnsetTxFeeRatePreference.
    - Added new GetTransaction rpc.
    
    GrpcWalletsService - Added new getTransaction gRPC boilerplate.
    
    CoreWalletsService - Added new getTransaction implementation.
    
    TxInfo - Added the new fields for displaying a tx summary from CLI.
    This is not intended to be more than a brief summary;  a block explorer
    or bitcoin-core client should be used to see the complete definition.
    
    TradeInfo - Added the new withdrawalTxId field defined in grpc.proto.
    
    CliMain - Added new 'case gettransaction'.
    
    TransactionFormat - Formats a TxInfo sent from the server to CLI.
    
    ColumnHeaderConstants - Added console headers used by TransactionFormat.
    
    TradeFormat - Displays a completed trade's WithdrawalTxId if present.
    
    Apitest - Adjusted affected tests: assert tx memo is persisted and
    test gettransaction.
    ghubstan committed Dec 14, 2020
    Configuration menu
    Copy the full SHA
    5522d0c View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    a0f1c22 View commit details
    Browse the repository at this point in the history
  3. Adjust create TransferwiseAccount test

    As per commit 88f26f9,
    do not autofill all currencies by default but keep all unselected.
    ghubstan committed Dec 14, 2020
    Configuration menu
    Copy the full SHA
    0384642 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    4be87a6 View commit details
    Browse the repository at this point in the history

Commits on Dec 15, 2020

  1. Configuration menu
    Copy the full SHA
    a341173 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2020

  1. Add new CoreApiExceptionHandler to gRPC services

    This change reduces gRPC service error handling duplication by moving
    it into a @singleton encapsulating everything needed to wrap
    an expected or unexpected core api exception into a gRPC
    StatusRuntimeException before sending it to the client.  It also
    fixes some boilerpate classes were gRPC error handling was missing.
    ghubstan committed Dec 16, 2020
    Configuration menu
    Copy the full SHA
    e6c6d3b View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1cd47fd View commit details
    Browse the repository at this point in the history
  3. Fix class level comment

    ghubstan committed Dec 16, 2020
    Configuration menu
    Copy the full SHA
    c60605f View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    f7c1103 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    2572e86 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    fa9ffa1 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2020

  1. Prevent excessive api calls

    This change provides a gRPC CallRateMeteringInterceptor to help protect
    the server and network against being overloaded by CLI scripting mistakes.
    
    An interceptor instance can be configured on a gRPC service to set
    individual method call rate limits on one or more of the the service's
    methods. For example, the GrpcOffersService could be configured with
    this interceptor to set the createoffer rate limit to 5/hour, and
    the takeoffer call rate limit could be set to 20/day.  Whenever a
    call rate limit is exceeded, the gRPC call is aborted and the client
    recieves a "rate limit exceeded" error.
    
    Below is a simple example showing how to set rate limits for one method
    in GrpcVersionService.
    
        final ServerInterceptor[] interceptors() {
            return new ServerInterceptor[]{
                    new CallRateMeteringInterceptor(new HashMap<>() {{
                        put("getVersion", new GrpcCallRateMeter(2, SECONDS));
                    }})
            };
        }
    
    It specifies a CLI can execute getversion 2 times / second.
    
    This is not a throttling mechansim, there is no blocking nor locking
    to slow call rates.  When call rates are exceeded, calls are
    simply aborted.
    ghubstan committed Dec 17, 2020
    Configuration menu
    Copy the full SHA
    2148a4d View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    89e2187 View commit details
    Browse the repository at this point in the history

Commits on Dec 19, 2020

  1. Configuration menu
    Copy the full SHA
    a5ed17e View commit details
    Browse the repository at this point in the history
  2. Make CallRateMeteringInterceptor configurable via json

    This adds a GrpcServiceRateMeteringConfig class that can read and
    write rate metering interceptor config files, and configure
    a gRPC rate metering service interceptor at startup.
    
    This seems excessive, but we need to be able to test and tune
    method rate metering without having to change hard coded, default
    interceptor rate meters.
    ghubstan committed Dec 19, 2020
    Configuration menu
    Copy the full SHA
    b307593 View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    5de910a View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    455ed67 View commit details
    Browse the repository at this point in the history
  5. Add license note

    ghubstan committed Dec 19, 2020
    Configuration menu
    Copy the full SHA
    830a5f0 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    9f679de View commit details
    Browse the repository at this point in the history
  7. Inject Config into CoreApi

    ghubstan committed Dec 19, 2020
    Configuration menu
    Copy the full SHA
    bb8d2ae View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    ea97a80 View commit details
    Browse the repository at this point in the history
  9. Configure GrpcVersionService's rate metering interceptor

    This change demonstrates how a method call rate metering interceptor
    is configured for a gRPC service.  GrpcVersionService uses a custom
    rate metering interceptor, or none.  A commented out, 'default'
    interceptor is defined as a usage example.
    ghubstan committed Dec 19, 2020
    Configuration menu
    Copy the full SHA
    87f75ee View commit details
    Browse the repository at this point in the history
  10. Add ApiTestConfig option --callRateMeteringConfigPath

    Points to a call rate metering interceptor configuration file.
    
    Test cases can build a config file, and the test harness will
    install it into a daemon's appDataDir before startup.
    The installed config file will be used to configure gRPC
    service rate metering interceptors.
    ghubstan committed Dec 19, 2020
    Configuration menu
    Copy the full SHA
    56a5c79 View commit details
    Browse the repository at this point in the history
  11. Install call rate metering config file before startup

    Copy the config file at --callRateMeteringConfigPath to each daemon's
    appDataDir, where it will be detected at server startup.
    ghubstan committed Dec 19, 2020
    Configuration menu
    Copy the full SHA
    d5657e9 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    fabd7c8 View commit details
    Browse the repository at this point in the history
  13. Configuration menu
    Copy the full SHA
    abc3940 View commit details
    Browse the repository at this point in the history
  14. Remove unused import

    ghubstan committed Dec 19, 2020
    Configuration menu
    Copy the full SHA
    a3eb4ed View commit details
    Browse the repository at this point in the history

Commits on Dec 21, 2020

  1. Configuration menu
    Copy the full SHA
    672eb79 View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    bdde24a View commit details
    Browse the repository at this point in the history
  3. Configuration menu
    Copy the full SHA
    64c2ac5 View commit details
    Browse the repository at this point in the history
  4. Configuration menu
    Copy the full SHA
    97dcac2 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    3a770f4 View commit details
    Browse the repository at this point in the history
  6. Configuration menu
    Copy the full SHA
    4aa4270 View commit details
    Browse the repository at this point in the history
  7. Configuration menu
    Copy the full SHA
    1507a2c View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    8ee3a15 View commit details
    Browse the repository at this point in the history
  9. Merge branch '09-refactor-grpc-error-handling' into 10-callrate-inter…

    …ceptor
    
    Adjust to reverting to reverting 6aa385e, and fix test file conflict.
    ghubstan committed Dec 21, 2020
    Configuration menu
    Copy the full SHA
    27efc5f View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2020

  1. Use a simpler, time windowing call rate meter

    Rewrote the GrpcCallRateMeter class and adjusted afected classes.
    
    These changes were requested in PR review
    bisq-network#4966 (review)
    ghubstan committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    01546ad View commit details
    Browse the repository at this point in the history
  2. Remove unused local var

    ghubstan committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    d615212 View commit details
    Browse the repository at this point in the history
  3. Remove redundant callCount field

    The size of the timestamp queue is the call count
    ghubstan committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    0d4ed95 View commit details
    Browse the repository at this point in the history
  4. Fix comment

    ghubstan committed Dec 22, 2020
    Configuration menu
    Copy the full SHA
    c8ef414 View commit details
    Browse the repository at this point in the history

Commits on Dec 23, 2020

  1. Support more fine grained rate metering

    We need to be able to define call rate meters for time spans not limited
    to TimeUnit intervals of 1 SECOND, 1 HOUR, or 1 DAY.  This change allows
    more flexibility, e.g., 10 per 5 seconds, 10 per 5 hrs, 100 per 7 days.
    ghubstan committed Dec 23, 2020
    Configuration menu
    Copy the full SHA
    6356476 View commit details
    Browse the repository at this point in the history
  2. Disable CallRateMeteringInterceptorTest and run it from test suite

    This will reduce the entire apitest suite's exec time
    ghubstan committed Dec 23, 2020
    Configuration menu
    Copy the full SHA
    b8c5a29 View commit details
    Browse the repository at this point in the history
  3. Fix GrpcCallRateMeter method and variable name

    - Change method isAllowed() -> checkAndIncrement().
    
    - Change variable allowedCallsPerTimeUnit -> allowedCallsPerTimeWindow.
    ghubstan committed Dec 23, 2020
    Configuration menu
    Copy the full SHA
    10727fc View commit details
    Browse the repository at this point in the history