Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sign transaction without broadcast #3475

Open
santiagorp opened this issue Mar 8, 2018 · 47 comments

Comments

Projects
None yet
@santiagorp
Copy link

commented Mar 8, 2018

How can we sign a transaction without broadcasting it to the network?

I could not find it in the docs and passing an EthereumTx object from ethereumjs-tx to the web3.eth.signTransaction method does not seem to work either.

We need to sign a transaction with metamask, send the signed transacion to our server and then broadcast it to the network.

The reason for this is that once that metamask prompts for signing a transacion the user can close the tab or navigate to a different location so the callback will never be invoked (we cannot know if the transaction was signed or not, mined, etc). As a result we do not have any way to store the txHash nor track the execution status.

In order to cover all the possible cases we need to sign the transaction with metamask, send it to our server and and then broadcast it from our own node. In this way we have total control of the status of the transaction.

@kumavis

This comment has been minimized.

Copy link
Member

commented Mar 10, 2018

We don't and won't support signing a transaction without submitting it, because it becomes a giant footgun in terms of nonce calculation inside MetaMask.

That said, we're very interested in giving better information over the state of a transaction. I think this can be best done by creating new RPC methods. Additionally I think its important to have a layer above transactions, so that the user can submit multiple transactions (e.g. retry at a higher price) without losing that reference for the dapp.

If you want to put together some RPC specs for what you'd like to see, I'd be happy to review, promote, and get implemented.

@santiagorp

This comment has been minimized.

Copy link
Author

commented Mar 10, 2018

Hi, thanks for replying. Some comments and proposals:

We don't and won't support signing a transaction without submitting it, because it becomes a giant footgun in terms of nonce calculation inside MetaMask.

Please take into account that the nonce is one of the parameters which can be provided to the transaction, so MetaMask has nothing to do with it in these cases. Maybe you could have an option to not broadcast the transaction when the nonce is one of the provided parameters. This does not enter in conflict your issues of nonce calculation and leaves the responsibility to the app developer.

In a high asynchronous environment where you have a server storing the app state in a DB and a web client broadcasting transactions to the blockchain you can never be sure when the transaction is going to be mined (if ever), so calculating the nonce in the web server and broadcasting it from our server is the only way to track the transaction status and to be sure that it is not submitted again (well, it can be submitted as many times as you want, but it will be mined only once since the nonce is the same). If we let MetaMask calculate the nonce, there is no reliable way to know if user will sign multiple times the same transaction by accident/bug/connection issues/etc. On the other hand we do not want to add expensive logic in the smart contract just to detect "duplicated" (two different transactions with the same data, signed at two different times) when it is clearly an UI/UX issue.

I guess that some of the problems you have with the nonce calculation are derived from this high asynchronous environment. For example a user signs a transaction using MetaMask and sets a low gas price, then he signs a different transaction with the same account using a different wallet with a very high gas price: the initial transaction from metamask will never get mined since the nonce was already used. So in my opinion, in these cases, it is the responsibility of the developer to take care of the nonce calculation and to keep track of the transaction status and not MetaMask.

Additionally I think its important to have a layer above transactions, so that the user can submit multiple transactions (e.g. retry at a higher price) without losing that reference for the dapp.

Yes, I agree, and again I think it is the responsibility of the app developer to contemplate all the possible outcomes and prepare the application for this. In our case one of the measures we have is to calculate the nonce ourselves: in the worst case the transaction is never mined (wrong nonce, so it is invalid) and it does not cost any gas to the end user.

If you want to put together some RPC specs for what you'd like to see, I'd be happy to review, promote, and get implemented.

This is a huge overhead just to track the status of a transaction. This added layer of complexity would be new source of problems (maintenance, bugs, etc). With the current status of MetaMask being open/shared between browser tabs and after page reloads/redirects (not sure if it is a chrome extension issue or a design feature), it is not possible.

We could create a transaction, encoded it properly and sign it using the web3.eth.personal.sign method, however the user will never know what he is really signing... so this is not an option.

@santiagorp santiagorp closed this Mar 10, 2018

@santiagorp santiagorp reopened this Mar 10, 2018

@santiagorp

This comment has been minimized.

Copy link
Author

commented Mar 10, 2018

I closed the issue by accident, please read my previous reply :-)

@santiagorp

This comment has been minimized.

Copy link
Author

commented Mar 12, 2018

Problem:

Question here: would this be a standard-problem (e.g.: many EthApp developers have this problem)

Yes, this affects to most Ethereum applications containing some client-server elements in their architecture (let's say hybrid DApps). This kind of architecture will stay due to multiple factors, but mainly related to specific requirements of the business model and technology not being mature enough in order to have completely decentralised architectures (related to privacy and security in decentralised databases).

On the other hand this is directly related to the kind of the end users: educated/blockchain enthusiasts or mainstream. If the intended public of the application is an educated user, then he knows what a confirmation is, what gas price/limit means, delays mining transactions due to network congestion, transactions not being relayed properly due to other network problems, node issues, etc. For these kind of users it is not a problem not to have certain measures which prevent users submitting several times the same transaction. We delegate the responsibility to the end user.

Just an example of a common scenario for this kind of power users:
An educated user signs a transaction (which will enable the smart contract to store some data and log something in an event). The transaction is unconfirmed or for whatever reason takes a long time to appear in a block explorer. He would probably wait a bit and check later on before doing it again, since he doesn't want to have two records in the smart contract with the same data.

In contrast, the average Joe would submit the transaction. Since he does not see any response (may be some WiFi reconnection), so he refreshes the page again and retries. Then he sees the word unconfirmed and then after waiting a while he retries again since nothing happened. In the end he will end up with 3 equal records. Two of them are unnecessary for him and he paid x3 what he needed.

The end users of our application are not familiar with blockchain technology and they are not (nor should!) aware of all its implications and details. Thus if we cannot track properly the transactions (by forcing nonce values and doing the broadcast ourselves) we cannot provide the user with a proper UI which prevents this kind of issues, restricts misuse or even abuse.

Currently once that a contract method has been invoked using MetaMask there are multiple scenarios in which the callback will never be executed:

  • User never signing/rejecting
  • User signing/rejecting after closing the app tab
  • User signing/rejecting after reloading the page
  • User signing/rejecting after navigating somewhere else

And even if it has being signed, and the callback invoked, there are no warranties that it has been relayed. In the best case we just have the txHash and just wait/listen for it.

Only if none of the above went wrong, then we would have a txHash. But this is still in the browser and it has to reach our server (hopefully no connection or server issues) so we can log it and track it. And even then there is still the possibility that the transaction is relayed and confirmed in a block before the txHash has been received on our server (due to network/connection issues as mentioned before), so matching an incoming event from the contracts it is extremely difficult if you don't have the txHash. Of course I could add always a nonce parameter to all my smart contract events and use that do identify my transactions... but this would contribute to bloat the blockchain just because MetaMask has this limitation. And it would be more expensive too!

Standard Solution

The only "standard" here is the web3 API and it allows to do this without issues. However you need full access to the private keys, which MetaMask have and the DApp don't, so it has to be MetaMask the one providing this functionality.

Suggested Solution:

  • MetaMask does not(!) send the TX, because of the set nonce (Dialogue)

I would not do it by default since it goes against the default behaviour of MetaMask and it would not maintain backwards compatibility. Just to my mind there are two options:

  • A configuration option for MetaMask which would allow setting the provider engine to not relay transactions if the nonce is provided. In this case the callback would return the rawTransaction.
  • The method web3.eth.signTransaction which is not part of the web3 API but from Metamask, might have an optional parameter to enable this behaviour.

MetaMask gets notified (tx success / nonce++) => how?

Since MetaMask is signing the transaction, MetaMask has the rawTransaction and can compute the txHash (keccak256 of the raw transaction) so you can listen for completion as usual. In the MetaMask UI the transaction would initially be flagged as Unseen, once seen by your nodes to Unconfirmed and finally to Confirmed.

Regarding the nonce, I am not sure about this: are you using the method eth.getTransactionCount(addr) or do you have any specific method? Otherwise you could just ignore unseen transactions for your calculations, only use the relayed ones. From your question, if you have problems to calculating the nonce for transactions not relayed by MetaMask I guess you still have the same problem if the user exports the key and signs transactions with that account in a different wallet provider, which means that the issue is not about NOT broadcasting the transactions but about having the private key signing transactions somewhere else. But in my case it would be much easier since you signed it and you have the txHash, so you can track it.

Remarks

Giving Devs Freedom/Control is usually a good thing. Bad devs will mess things up, but good devs will use the freedom/control to make good apps! Restraining good devs unnecessarily should be avoided. Keeping bad devs under control, at the cost of restraining good devs => should be avoided.

Completely agree with the above.

Thanks for your time and effort!

@santiagorp

This comment has been minimized.

Copy link
Author

commented Mar 13, 2018

Thanks a lot for your help. I believe this is an essential feature if we want to develop applications with better usability for mainstream users.

@kingflurkel

This comment has been minimized.

Copy link

commented Mar 28, 2018

If we want to successfully complete our Gas Station front end build, we really need this too. Thanks for taking dApp devs into consideration ♡

@sponnet

This comment has been minimized.

Copy link
Contributor

commented Mar 28, 2018

We're also in need for this functionality. We're developing a GasStation , where users can send ERC20 tokens around without the need of having gas. The GasStationService needs a signed transaction ( an approval ) to work. It would be great if

  • metamask would allow this
  • you would accept a PR from us that enables it ( like implemented above )

https://github.com/EthereumGasStation/ethereumgasstationserver/blob/master/test/test-api.js#L97-L113

@sponnet

This comment has been minimized.

Copy link
Contributor

commented Mar 28, 2018

Oh - and we would be the first users to implement ethereum/EIPs#712 !

@pizza-r0b

This comment has been minimized.

Copy link

commented Apr 5, 2018

Anyone know when this will be implemented? I have a use case for this and would love to be able to use web3.eth.signTransaction with meta mask.

@pizza-r0b

This comment has been minimized.

Copy link

commented Apr 6, 2018

@lazaridiscom happy to provide an example. This NEEDS to be implemented if we want to build more powerful decentralized applications (with meta mask) that don't just rely on client-side interactions.

@martirosyan-kar

This comment has been minimized.

Copy link

commented May 3, 2018

Hey @lazaridiscom. We also are in need of this feature.
Are there any estimates?
We need this feature too much and our team of Nodejs/ReactJS team is willing to help you guys with the implementation.

@martirosyan-kar

This comment has been minimized.

Copy link

commented May 3, 2018

@pizza-r0b Can you please provide some details?

@jeffersoncarpenter

This comment has been minimized.

Copy link

commented May 7, 2018

I need this feature as well. I am developing an online store accepting ERC20 tokens as payments. Customers will sign transactions sending tokens to our wallet, send our server the txHash, and then allow the transaction to be broadcast to the network. This enables our server to know with 100% certainty who sent what payment.

@pizza-r0b if you found a way to sign a transaction without sending it through MetaMask's web3 api, please share what it is.

@pizza-r0b

This comment has been minimized.

Copy link

commented May 7, 2018

@martirosyan-kar @jeffersoncarpenter - sounds like your use cases may differ from mine. @jeffersoncarpenter this may work for you if all you want to do it verify a user is who they say they are. I verify the users address by signing a message and sending the hash to the server, then I call my contract method, but use encodeABI to get the encoded ABI byte code. At this point I have verified the user is who they say they are and have dynamically generated a function call on the server. I then send the encoded ABI byte code back to the client for the user to execute the transaction. There is still a chance the user may not execute the transaction, but I think by listening to contract events and potentially running jobs at different times problems that arise from users not executing transactions could be mitigated. However, it would still be nice to be able to execute a signed transaction on the server. This is just a work around for my use case.

@julien51

This comment has been minimized.

Copy link

commented May 17, 2018

This would also be fantastic for us. For reasons similar to @santiagorp we need to trigger transactions from the backend. It would be elegant to split the "signing" from the "sending" and could also be very useful in the context of ERC725.

@pizza-r0b

This comment has been minimized.

Copy link

commented May 23, 2018

This would be very helpful in the case that the server needs to react to a successful transaction. Without this I am relying on now is encodeAbi (executed client side) and events emitted from the contract to react to certain things that have happened. This leads to a problem - what if the server is down and the emitted event is missed? I can run some kind of scheduled job to do something, but what if the client holds onto the encoded ABI and executes the transaction after the scheduled job is run? I am going to try to add a date argument to the solidity function and check if now - date < 1 day, but I don't know yet if now will be encoded into the transaction when I call encodedAbi

@qjpcpu

This comment has been minimized.

Copy link

commented Jun 14, 2018

i need this feature too

@rstormsf

This comment has been minimized.

Copy link

commented Jun 25, 2018

+1

The reason why I need this feature because metamask likes to fail if it didn't get a receipt within 50 blocks.
I'd like to signTx and broadcast on my own

OR

allow metamask to provide my own broadcaster fn as a parameter.

OR
Lets provide some customizable parameters for:

  1. Poll interval for receipt (5-10 sec)
  2. Max interval count to poll for receipt (default 50, but allow devs to provide their own)

@kumavis

@ldub

This comment has been minimized.

Copy link

commented Jun 25, 2018

+1 also need this feature

I've got a backend system updating its own metadata storage after the tx goes through, but in the case of a network partition or other failure after the metamask interaction but before the backend recieves the resulting txHash to watch, the metadata is never updated and the system ends up in a corrupt state. Fixing the metadata manually after such a failure is a huge pain.

The best way to solve this is to have the backend broadcast the transaction.

@jerryhorak

This comment has been minimized.

Copy link

commented Jul 12, 2018

+1

Waiting for this as well. I have a need to send transactions from the backend after being signed client side. Currently, the solution for my use case is less than ideal right now.. optimally, I'd like my app to generate a raw transaction on the server, deliver to the user to sign client side, and then send signed transaction to the node from my backend services (nonce is without concern and taken into account).

Edit: This is for a private network (potential consortium, if you will), if that makes it more relevant. The idea is to manage transactions through the interface with high accuracy (a la @ldub). Transactions outside the interface is beyond scope.

@eswak

This comment has been minimized.

Copy link

commented Jul 18, 2018

+1

We also need this feature, signing a transaction from metamask & getting the raw transaction hex string as a callback would allow for better tracking of the operation on our side.

Also, if you want to provide different signing mechanisms for a tx in a web page (think: metamask, ledger hardware wallet, ...), it is a pain to not have control on the broadcasting because some signing mechanisms will do the broadcast for you and some will not. The workaround that I found for now is to broadcast the transaction client-side (like metamask does, through the provided web3) to get an homogeneous user experience with other signing mechanisms, but that doesn't allow for a proper tracking of pending transactions if the page is refreshed or if the transaction is signed after the webapp has been closed.

@ghost

This comment has been minimized.

Copy link

commented Aug 14, 2018

I guess everyone needs this feature. We find it very hard to use Metamask while our DAPP needs to sign a transaction in order to keep track of the state of the contract in "real time".

@rickygv99

This comment has been minimized.

Copy link

commented Sep 5, 2018

+1 We also need this feature. We're developing an online payment store with a few blockchain transactions per payment instance. The transactions must be sequential and consequently the client must wait until the transactions are mined as is. With this feature, we will be able to queue the transactions on a server - it would be immensely helpful to us.

@andrewsource147

This comment has been minimized.

Copy link

commented Sep 7, 2018

+1 , Really exciting for this feature, Metamask will be like my coldwallet. We build a website that users can access from many kind of wallets. Work flow when user access Metamask is quite different from others.

@sargonpiraev

This comment has been minimized.

Copy link

commented Sep 14, 2018

+1
This feature will help a lot. In order to use same sign transaction flow for all auth methods in app (trezor, ledger, web3 account) we need this feature. Because we want to send signed transaction by our side.

@sidsverma

This comment has been minimized.

Copy link

commented Sep 27, 2018

+1
This will help me improve the UX of my dapp!

@kowalski

This comment has been minimized.

Copy link

commented Oct 4, 2018

+1 for this feature request.

If I may throw in a suggestion I encountered personal_signTransaction method described here: https://github.com/paritytech/wiki/blob/master/JSONRPC-personal-module.md#personal_signtransaction

This is part of Personal module, which I think is not an official standard, nevertheless is already partially implemented by Metamask (with personal_sign).

MetaMask gets notified (tx success / nonce++) => how?

I would personally recommend against notyfing Metamask about nonce increase. I think Metamask should simply check internal nonce counter and compare it with result of eth_getTransactionCount(address, 'pending') and take max(..) of these two.
I agree there is timing issue, where the transaction submitted to a third-party node has not yet been spotted by Infura and as a result Metamask would suggest already taken nonce.

@HenryTimelessness

This comment has been minimized.

Copy link

commented Oct 5, 2018

+1 this will be great for many dapps

@junw3i

This comment has been minimized.

Copy link

commented Oct 5, 2018

+1 in need of this

@xpepermint

This comment has been minimized.

Copy link

commented Oct 10, 2018

+1

@piotrwawrzyn

This comment has been minimized.

Copy link

commented Oct 15, 2018

Is this implemented already?

@bmlis

This comment has been minimized.

Copy link

commented Oct 15, 2018

+1, right now because of the default behaviour I have to abandon metamask in some of my projects.

@sidsverma

This comment has been minimized.

Copy link

commented Oct 16, 2018

I have created a repo which signs a transaction and can be used to send it to the backend via an api and then broadcast(relay) it via your backend onto the blockchain.
It might be helpful for you: https://github.com/sidsverma/eth_signTypedData-example
I have used metamask v0.20.7 and used eth_signTypedData.

@jpthor

This comment has been minimized.

Copy link

commented Oct 26, 2018

Paging @kumavis

We need this feature too. It will massively improve the user experience of our dApp by allowing use to query the tx before submitting to our nodes; especially for multiple consequential transactions. We can receive all the signatures and process the tx asynchronously to the user - to them the transactions will appear practically instantly.

Currently it is untenable to think u have to wait until the tx is processed on-chain before doing the next action. Once our server has the signed and valid transactions then that is all that is needed to send the user to the next action, instead of waiting around for the tx to be mined

@danfinlay

This comment has been minimized.

Copy link
Contributor

commented Nov 6, 2018

@jpthor

Sounds like your use case would also be satisfied if MetaMask let you request the final tx after submission, instead of waiting until it is mined. Is this correct?

@jpthor

This comment has been minimized.

Copy link

commented Nov 7, 2018

@danfinlay
Hey thanks for getting back to me! I caught up with the Metamask team in DevCon and they said they'd bump this priority up (🙏)

Yep - if our frontend can collect the signed tx's off MM (before or after broadcast - doesn't matter) then we can assume the TX will get mined (eventually) and we can move the user on to the next part of the user experience asynchronously (with a payment pending flag). Then later we can ping the blockchain to observe the completed TX to update the flag to "payment completed".

@miguelmota

This comment has been minimized.

Copy link

commented Nov 7, 2018

My application requires the signed user transactions to be processed by the backend, so this feature is definitely needed

@MoMannn

This comment has been minimized.

Copy link

commented Nov 29, 2018

+1

@tomonari-t

This comment has been minimized.

Copy link

commented Jan 25, 2019

+1 Really want this feature. If we use this future, at the backend, we can control txs

@Perseverance

This comment has been minimized.

Copy link

commented Feb 13, 2019

+1 We need this because our backend broadcasts the transaction

@thcrnk

This comment has been minimized.

Copy link

commented Feb 13, 2019

+1

@NemboKid

This comment has been minimized.

Copy link

commented Feb 27, 2019

Many dapps would really benefit from this!

@epiqueras

This comment has been minimized.

Copy link

commented Mar 11, 2019

+1. In my use case, I need to send a signed transaction as authentication for storing some data in a server without requiring an extra MetaMask popup. I need to do it before the transaction is broadcasted so someone doesn't grab it and beat the user to it.

@guenoledc

This comment has been minimized.

Copy link

commented Mar 25, 2019

+1.
Working on a workaround on my node to intercept the metamask sendRawTransaction call to be able to submit it later when I want

@olaf89

This comment has been minimized.

Copy link

commented Mar 30, 2019

This is definitely needed

@fare

This comment has been minimized.

Copy link

commented May 2, 2019

+1e1000

Without this feature, any DApp more complex than payment can be killed by a targeted DDoS attack and cause the user to lose all his money. The only defense is to have redundant backend servers (behind TOR and/or in Akamai-grade datacenters) to ensure delivery, and this requires persisting the raw tx on the private distributed database BEFORE anything is sent to the public network. Also, requiring the user to sign the same transaction multiple times is a huge UX vulnerability, an invitation to attackers baiting him into signing multiple different transactions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.