Sign transaction without broadcast

[santiagorp]

How can we sign a transaction without broadcasting it to the network?

I could not find it in the docs and passing an EthereumTx object from ethereumjs-tx to the web3.eth.signTransaction method does not seem to work either.

We need to sign a transaction with metamask, send the signed transacion to our server and then broadcast it to the network.

The reason for this is that once that metamask prompts for signing a transacion the user can close the tab or navigate to a different location so the callback will never be invoked (we cannot know if the transaction was signed or not, mined, etc). As a result we do not have any way to store the txHash nor track the execution status.

In order to cover all the possible cases we need to sign the transaction with metamask, send it to our server and and then broadcast it from our own node. In this way we have total control of the status of the transaction.

[kumavis]

We don't and won't support signing a transaction without submitting it, because it becomes a giant footgun in terms of nonce calculation inside MetaMask.

That said, we're very interested in giving better information over the state of a transaction. I think this can be best done by creating new RPC methods. Additionally I think its important to have a layer above transactions, so that the user can submit multiple transactions (e.g. retry at a higher price) without losing that reference for the dapp.

If you want to put together some RPC specs for what you'd like to see, I'd be happy to review, promote, and get implemented.

[santiagorp]

Hi, thanks for replying. Some comments and proposals:

We don't and won't support signing a transaction without submitting it, because it becomes a giant footgun in terms of nonce calculation inside MetaMask.

Please take into account that the nonce is one of the parameters which can be provided to the transaction, so MetaMask has nothing to do with it in these cases. Maybe you could have an option to not broadcast the transaction when the nonce is one of the provided parameters. This does not enter in conflict your issues of nonce calculation and leaves the responsibility to the app developer.

In a high asynchronous environment where you have a server storing the app state in a DB and a web client broadcasting transactions to the blockchain you can never be sure when the transaction is going to be mined (if ever), so calculating the nonce in the web server and broadcasting it from our server is the only way to track the transaction status and to be sure that it is not submitted again (well, it can be submitted as many times as you want, but it will be mined only once since the nonce is the same). If we let MetaMask calculate the nonce, there is no reliable way to know if user will sign multiple times the same transaction by accident/bug/connection issues/etc. On the other hand we do not want to add expensive logic in the smart contract just to detect "duplicated" (two different transactions with the same data, signed at two different times) when it is clearly an UI/UX issue.

I guess that some of the problems you have with the nonce calculation are derived from this high asynchronous environment. For example a user signs a transaction using MetaMask and sets a low gas price, then he signs a different transaction with the same account using a different wallet with a very high gas price: the initial transaction from metamask will never get mined since the nonce was already used. So in my opinion, in these cases, it is the responsibility of the developer to take care of the nonce calculation and to keep track of the transaction status and not MetaMask.

Additionally I think its important to have a layer above transactions, so that the user can submit multiple transactions (e.g. retry at a higher price) without losing that reference for the dapp.

Yes, I agree, and again I think it is the responsibility of the app developer to contemplate all the possible outcomes and prepare the application for this. In our case one of the measures we have is to calculate the nonce ourselves: in the worst case the transaction is never mined (wrong nonce, so it is invalid) and it does not cost any gas to the end user.

If you want to put together some RPC specs for what you'd like to see, I'd be happy to review, promote, and get implemented.

This is a huge overhead just to track the status of a transaction. This added layer of complexity would be new source of problems (maintenance, bugs, etc). With the current status of MetaMask being open/shared between browser tabs and after page reloads/redirects (not sure if it is a chrome extension issue or a design feature), it is not possible.

We could create a transaction, encoded it properly and sign it using the web3.eth.personal.sign method, however the user will never know what he is really signing... so this is not an option.

[santiagorp]

I closed the issue by accident, please read my previous reply :-)

[santiagorp]

Problem:

Question here: would this be a standard-problem (e.g.: many EthApp developers have this problem)

Yes, this affects to most Ethereum applications containing some client-server elements in their architecture (let's say hybrid DApps). This kind of architecture will stay due to multiple factors, but mainly related to specific requirements of the business model and technology not being mature enough in order to have completely decentralised architectures (related to privacy and security in decentralised databases).

On the other hand this is directly related to the kind of the end users: educated/blockchain enthusiasts or mainstream. If the intended public of the application is an educated user, then he knows what a confirmation is, what gas price/limit means, delays mining transactions due to network congestion, transactions not being relayed properly due to other network problems, node issues, etc. For these kind of users it is not a problem not to have certain measures which prevent users submitting several times the same transaction. We delegate the responsibility to the end user.

Just an example of a common scenario for this kind of power users:
An educated user signs a transaction (which will enable the smart contract to store some data and log something in an event). The transaction is unconfirmed or for whatever reason takes a long time to appear in a block explorer. He would probably wait a bit and check later on before doing it again, since he doesn't want to have two records in the smart contract with the same data.

In contrast, the average Joe would submit the transaction. Since he does not see any response (may be some WiFi reconnection), so he refreshes the page again and retries. Then he sees the word unconfirmed and then after waiting a while he retries again since nothing happened. In the end he will end up with 3 equal records. Two of them are unnecessary for him and he paid x3 what he needed.

The end users of our application are not familiar with blockchain technology and they are not (nor should!) aware of all its implications and details. Thus if we cannot track properly the transactions (by forcing nonce values and doing the broadcast ourselves) we cannot provide the user with a proper UI which prevents this kind of issues, restricts misuse or even abuse.

Currently once that a contract method has been invoked using MetaMask there are multiple scenarios in which the callback will never be executed:

• User never signing/rejecting
• User signing/rejecting after closing the app tab
• User signing/rejecting after reloading the page
• User signing/rejecting after navigating somewhere else

And even if it has being signed, and the callback invoked, there are no warranties that it has been relayed. In the best case we just have the txHash and just wait/listen for it.

Only if none of the above went wrong, then we would have a txHash. But this is still in the browser and it has to reach our server (hopefully no connection or server issues) so we can log it and track it. And even then there is still the possibility that the transaction is relayed and confirmed in a block before the txHash has been received on our server (due to network/connection issues as mentioned before), so matching an incoming event from the contracts it is extremely difficult if you don't have the txHash. Of course I could add always a nonce parameter to all my smart contract events and use that do identify my transactions... but this would contribute to bloat the blockchain just because MetaMask has this limitation. And it would be more expensive too!

Standard Solution

The only "standard" here is the web3 API and it allows to do this without issues. However you need full access to the private keys, which MetaMask have and the DApp don't, so it has to be MetaMask the one providing this functionality.

Suggested Solution:

• MetaMask does not(!) send the TX, because of the set nonce (Dialogue)

I would not do it by default since it goes against the default behaviour of MetaMask and it would not maintain backwards compatibility. Just to my mind there are two options:

• A configuration option for MetaMask which would allow setting the provider engine to not relay transactions if the nonce is provided. In this case the callback would return the rawTransaction.
• The method web3.eth.signTransaction which is not part of the web3 API but from Metamask, might have an optional parameter to enable this behaviour.

MetaMask gets notified (tx success / nonce++) => how?

Since MetaMask is signing the transaction, MetaMask has the rawTransaction and can compute the txHash (keccak256 of the raw transaction) so you can listen for completion as usual. In the MetaMask UI the transaction would initially be flagged as Unseen, once seen by your nodes to Unconfirmed and finally to Confirmed.

Regarding the nonce, I am not sure about this: are you using the method eth.getTransactionCount(addr) or do you have any specific method? Otherwise you could just ignore unseen transactions for your calculations, only use the relayed ones. From your question, if you have problems to calculating the nonce for transactions not relayed by MetaMask I guess you still have the same problem if the user exports the key and signs transactions with that account in a different wallet provider, which means that the issue is not about NOT broadcasting the transactions but about having the private key signing transactions somewhere else. But in my case it would be much easier since you signed it and you have the txHash, so you can track it.

Remarks

Giving Devs Freedom/Control is usually a good thing. Bad devs will mess things up, but good devs will use the freedom/control to make good apps! Restraining good devs unnecessarily should be avoided. Keeping bad devs under control, at the cost of restraining good devs => should be avoided.

Completely agree with the above.

Thanks for your time and effort!

[santiagorp]

Thanks a lot for your help. I believe this is an essential feature if we want to develop applications with better usability for mainstream users.

[kingflurkel]

If we want to successfully complete our Gas Station front end build, we really need this too. Thanks for taking dApp devs into consideration ♡

[sponnet]

We're also in need for this functionality. We're developing a GasStation , where users can send ERC20 tokens around without the need of having gas. The GasStationService needs a signed transaction ( an approval ) to work. It would be great if

• metamask would allow this
• you would accept a PR from us that enables it ( like implemented above )

https://github.com/EthereumGasStation/ethereumgasstationserver/blob/master/test/test-api.js#L97-L113

[sponnet]

Oh - and we would be the first users to implement ethereum/EIPs#712 !

[pizzarob]

Anyone know when this will be implemented? I have a use case for this and would love to be able to use web3.eth.signTransaction with meta mask.

[pizzarob]

@lazaridiscom happy to provide an example. This NEEDS to be implemented if we want to build more powerful decentralized applications (with meta mask) that don't just rely on client-side interactions.

[martirosyan-kar]

Hey @lazaridiscom. We also are in need of this feature.
Are there any estimates?
We need this feature too much and our team of Nodejs/ReactJS team is willing to help you guys with the implementation.

[martirosyan-kar]

@pizza-r0b Can you please provide some details?

[negatratoron]

I need this feature as well. I am developing an online store accepting ERC20 tokens as payments. Customers will sign transactions sending tokens to our wallet, send our server the txHash, and then allow the transaction to be broadcast to the network. This enables our server to know with 100% certainty who sent what payment.

@pizza-r0b if you found a way to sign a transaction without sending it through MetaMask's web3 api, please share what it is.

[pizzarob]

@martirosyan-kar @jeffersoncarpenter - sounds like your use cases may differ from mine. @jeffersoncarpenter this may work for you if all you want to do it verify a user is who they say they are. I verify the users address by signing a message and sending the hash to the server, then I call my contract method, but use encodeABI to get the encoded ABI byte code. At this point I have verified the user is who they say they are and have dynamically generated a function call on the server. I then send the encoded ABI byte code back to the client for the user to execute the transaction. There is still a chance the user may not execute the transaction, but I think by listening to contract events and potentially running jobs at different times problems that arise from users not executing transactions could be mitigated. However, it would still be nice to be able to execute a signed transaction on the server. This is just a work around for my use case.

[julien51]

This would also be fantastic for us. For reasons similar to @santiagorp we need to trigger transactions from the backend. It would be elegant to split the "signing" from the "sending" and could also be very useful in the context of ERC725.

[pizzarob]

This would be very helpful in the case that the server needs to react to a successful transaction. Without this I am relying on now is encodeAbi (executed client side) and events emitted from the contract to react to certain things that have happened. This leads to a problem - what if the server is down and the emitted event is missed? I can run some kind of scheduled job to do something, but what if the client holds onto the encoded ABI and executes the transaction after the scheduled job is run? I am going to try to add a date argument to the solidity function and check if now - date < 1 day, but I don't know yet if now will be encoded into the transaction when I call encodedAbi

[qjpcpu]

i need this feature too

[rstormsf]

+1

The reason why I need this feature because metamask likes to fail if it didn't get a receipt within 50 blocks.
I'd like to signTx and broadcast on my own

OR

allow metamask to provide my own broadcaster fn as a parameter.

OR
Lets provide some customizable parameters for:

1. Poll interval for receipt (5-10 sec)
2. Max interval count to poll for receipt (default 50, but allow devs to provide their own)

@kumavis

[ldub]

+1 also need this feature

I've got a backend system updating its own metadata storage after the tx goes through, but in the case of a network partition or other failure after the metamask interaction but before the backend recieves the resulting txHash to watch, the metadata is never updated and the system ends up in a corrupt state. Fixing the metadata manually after such a failure is a huge pain.

The best way to solve this is to have the backend broadcast the transaction.

[jhorak01]

+1

Waiting for this as well. I have a need to send transactions from the backend after being signed client side. Currently, the solution for my use case is less than ideal right now.. optimally, I'd like my app to generate a raw transaction on the server, deliver to the user to sign client side, and then send signed transaction to the node from my backend services (nonce is without concern and taken into account).

Edit: This is for a private network (potential consortium, if you will), if that makes it more relevant. The idea is to manage transactions through the interface with high accuracy (a la @ldub). Transactions outside the interface is beyond scope.

[eswak]

+1

We also need this feature, signing a transaction from metamask & getting the raw transaction hex string as a callback would allow for better tracking of the operation on our side.

Also, if you want to provide different signing mechanisms for a tx in a web page (think: metamask, ledger hardware wallet, ...), it is a pain to not have control on the broadcasting because some signing mechanisms will do the broadcast for you and some will not. The workaround that I found for now is to broadcast the transaction client-side (like metamask does, through the provided web3) to get an homogeneous user experience with other signing mechanisms, but that doesn't allow for a proper tracking of pending transactions if the page is refreshed or if the transaction is signed after the webapp has been closed.

[ghost]

I guess everyone needs this feature. We find it very hard to use Metamask while our DAPP needs to sign a transaction in order to keep track of the state of the contract in "real time".

[piotrwawrzyn]

Is this implemented already?

[bmlis]

+1, right now because of the default behaviour I have to abandon metamask in some of my projects.

[sidsverma]

I have created a repo which signs a transaction and can be used to send it to the backend via an api and then broadcast(relay) it via your backend onto the blockchain.
It might be helpful for you: https://github.com/sidsverma/eth_signTypedData-example
I have used metamask v0.20.7 and used eth_signTypedData.

[jpthor]

Paging @kumavis

We need this feature too. It will massively improve the user experience of our dApp by allowing use to query the tx before submitting to our nodes; especially for multiple consequential transactions. We can receive all the signatures and process the tx asynchronously to the user - to them the transactions will appear practically instantly.

Currently it is untenable to think u have to wait until the tx is processed on-chain before doing the next action. Once our server has the signed and valid transactions then that is all that is needed to send the user to the next action, instead of waiting around for the tx to be mined

[danfinlay]

@jpthor

Sounds like your use case would also be satisfied if MetaMask let you request the final tx after submission, instead of waiting until it is mined. Is this correct?

[jpthor]

@danfinlay
Hey thanks for getting back to me! I caught up with the Metamask team in DevCon and they said they'd bump this priority up (🙏)

Yep - if our frontend can collect the signed tx's off MM (before or after broadcast - doesn't matter) then we can assume the TX will get mined (eventually) and we can move the user on to the next part of the user experience asynchronously (with a payment pending flag). Then later we can ping the blockchain to observe the completed TX to update the flag to "payment completed".

[miguelmota]

My application requires the signed user transactions to be processed by the backend, so this feature is definitely needed

[MoMannn]

+1

[tomonari-t]

+1 Really want this feature. If we use this future, at the backend, we can control txs

[Perseverance]

+1 We need this because our backend broadcasts the transaction

[thcrnk]

+1

[NemboKid]

Many dapps would really benefit from this!

[epiqueras]

+1. In my use case, I need to send a signed transaction as authentication for storing some data in a server without requiring an extra MetaMask popup. I need to do it before the transaction is broadcasted so someone doesn't grab it and beat the user to it.

[guenoledc]

+1.
Working on a workaround on my node to intercept the metamask sendRawTransaction call to be able to submit it later when I want

[olaf89]

This is definitely needed

[fare]

+1e1000

Without this feature, any DApp more complex than payment can be killed by a targeted DDoS attack and cause the user to lose all his money. The only defense is to have redundant backend servers (behind TOR and/or in Akamai-grade datacenters) to ensure delivery, and this requires persisting the raw tx on the private distributed database BEFORE anything is sent to the public network. Also, requiring the user to sign the same transaction multiple times is a huge UX vulnerability, an invitation to attackers baiting him into signing multiple different transactions.

[araa47]

Hi, this feature is definitely needed for a layer 2 solution. I understand this is a security concern but there are proposals for this too, by using application specific wallet accounts. ethereum/EIPs#1775 .

Basically in a layer2 solution, once you enter layer2 by locking your funds in a smart contract, you don't want to sign and broadcast transactions to the ethereum node. Instead you want to be sending these messages to the operator of the layer 2 solution. Thus there should be a feature making this possible.

I agree that this shouldn't be possible for the main account since it is a security concern, however this is something that will be needed to continue building applications in the ethereum network that can be highly scaled.

[john-osullivan]

I want to leverage Metamask as a signer for Dapps on multiple chains -- having transaction signing directly tied to transaction broadcasting makes that kind of impossible. Eagerly awaiting an answer to this issue!!

[danfinlay]

Hi, this feature is definitely needed for a layer 2 solution. I understand this is a security concern but there are proposals for this too, by using application specific wallet accounts. ethereum/EIPs#1775 .

Hi, I'm a MetaMask developer and co-author of the EIP 1775 that you referenced.

We wrote EIP 1775 in part to help MetaMask address the issues of layer-2 or other-chain message signing, and in a way that avoids the type of painful user experiences that have been caused when interfaces were able to influence the user nonce in the past. (Not just security, as suggested a few times above)

If we ever do allow apps to edit nonce, or sign messages without an immediate broadcast, it would need to be with heavy warnings that basically ensured only developers (or people who understood the implications) could use it. Maybe a nonce comprehension quiz, I don't know. What I do know is that letting dapp developers edit normal users' nonces put undue burden on our support systems in the past, and created a lot of user grief, and that's why you probably won't see fast progress on this issue, but will see more likely progress on features that involve the creation of new keys that are permitted to a domain.

I recently had a conversation with remco of 0x that I think may result in an even simpler app-keys API too, so hopefully I'll have some time to write that out soon.

Sorry for the wait, we know there are a lot of exciting ways for MetaMask to unlock developer creativity, and that's what we're focused on.

[fare]

So basically you're saying MetaMask is totally inappropriate for either lay users (totally confusing UI wrt retrying) OR app developers (totally insecure for any layer 2 usage)? What's the whole point of MetaMask at all, then? Who's the audience? Looks like you're making compromises in the hope of pleasing everyone and end up pleasing no one at all.

If your target audience is developers whose creativity you want to unlock, then add the necessary hooks for us to write secure applications, which implies controlling nonces and saving signed txs to multiple datacenters before sending them on public networks.

If your target audience is end-users, then remove any and all developer support and focus on making a secure wallet with usable experience with respect to retrying. Support payments only, for ethers and a curated whitelist of ERC20, ERC721, ERC777 tokens, and other authorized apps, and drop any support for developer-defined apps.

The two just don't and can't possibly mix, though they can share some underlying libraries (like web3.js, which also has its problems).

For a slightly more configurable use—let the user decide which apps he trusts to issue transactions for which account. And ensure only one app or plugin will handle retrying for each account.

[danfinlay]

So basically you're saying MetaMask is totally inappropriate for either lay users (totally confusing UI wrt retrying) OR app developers (totally insecure for any layer 2 usage)?

No, I'm not saying that at all. We can both improve base-layer retry UX and also provide secure ways of managing L2 innovation.

If your target audience is developers whose creativity you want to unlock, then add the necessary hooks for us to write secure applications

We are, proposals like ethereum/EIPs#1775, among other current proposals we are still drafting. By allowing applications to freely manage keys within a permitted scope, we can allow developer freedom of nonce management while also protecting the basic user expectations of not spontaneously losing funds from their primary accounts.

If your target audience is end-users, then remove any and all developer support and focus on making a secure wallet with usable experience with respect to retrying.

What? No, we won't remove developer support. We will continue to improve the way we inform users about the state of transactions and how we advise them to proceed, though.

For a slightly more configurable use—let the user decide which apps he trusts to issue transactions for which account.

This is basically one of the things I said in my post. I think we agree more than you think we do. To quote myself:

If we ever do allow apps to edit nonce, or sign messages without an immediate broadcast, it would need to be with heavy warnings that basically ensured only developers (or people who understood the implications) could use it.

This is me saying "Maybe we can permit this, but we need to find a way of making the permission comprehensible". Or at least scary to non-expert users.

Anyways, I was just trying to clarify that there are other ways of providing layer-2 scaling without this feature, and making more clear what this feature would need to be integrated.

We definitely want L2 in MetaMask, and we want it to be usable by normal people.

[fare]

OK, that's a better story, but I believe you can do much better:

1. Intercepting messages to persist them before sending is not just a matter of layer 2 "scaling". It's any reliable layer 2 at all. If you want to survive device crash, DDoS attacks, disk failures, stolen device, broken internet connection, etc., you MUST commit your transactions to replicated DB before you issue them on the public internet. That's also necessary for simple payments, if you want to make sure you spend once and only once in the face of sometimes necessary retries.

2. Being able to track your persistent data from your configured remote DB servers is essential to present a UI to users as to what messages were issued, how they compare to what messages are pending or confirmed on the blockchain, when to retry, and how to cancel (if still affordable).

3. The configuration for remote DB servers could itself be stored encrypted on the blockchain, so users can easily retrieve it from their master key should their master device have to be replaced.

4. The above can be done by plugins that are wholly independent of the application. Note that the application will have to store its own application data in the very same DB servers. Users in general want to preserve the data of applications they disabled, but may want to explicitly purge the associated data after a while.

5. While multiple accounts is fine and dandy, what is more important and more useful is a limit on how much each application can draw from the current account, so users can track spending (default, 0). Just like temporary credit cards linked to a master credit card, with amount and time limits. Preserving the account and its identity is sometimes a requirement for behavior across applications. Creating multiple accounts for anonymity is fine, but cuts across support for multiple applications.

[danfinlay]

Intercepting messages to persist them before sending is not just a matter of layer 2 "scaling". It's any reliable layer 2 at all. If you want to survive device crash, DDoS attacks, disk failures, stolen device, broken internet connection, etc., you MUST commit your transactions to replicated DB before you issue them on the public internet.

That's a valid concern, but I don't agree that this issue ("Sign transaction without broadcast") is the one way to get a solution to it, and it introduces dangers to users in the form of impossible nonce-management as we've explained above.

For example (of how your items 1 and 2 could be solved otherwise), since MetaMask already does persist outgoing messages before sending them, your concern here could be addressed by simply providing an API to review the transactions that have been sent from your domain through the current MetaMask instance. This would seem like a pretty safe change that satisfies one of your concerns here.

The configuration for remote DB servers could itself be stored encrypted on the blockchain, so users can easily retrieve it from their master key should their master device have to be replaced.

This is another good example of something solved by the app keys proposal EIP 1775, which we're making good progress on.

I'm unclear what number 4 implies here.

While multiple accounts is fine and dandy, what is more important and more useful is a limit on how much each application can draw from the current account, so users can track spending (default, 0). Just like temporary credit cards linked to a master credit card, with amount and time limits. Preserving the account and its identity is sometimes a requirement for behavior across applications. Creating multiple accounts for anonymity is fine, but cuts across support for multiple applications.

I couldn't agree more, and this is initially provided by the token approve() method, and we are continuing to pursue allowance-based permissions to simplify logins in the future. However, (getting back on topic) there is nothing about an allowance that requires that the website control when MetaMask broadcasts a transaction.

If implementing this feature made some new things possible safely, we would be eagerly considering it, but it seems clear that safety is off the table, and all the justifications that continue rolling in seem like they could be addressed as their own issues, with their own safe solutions.

To others in this thread: If you have a backend server and it needs to send transactions, I recommend you load it with your own gas, manage your own nonce, and send transactions with it. If you want your contracts to treat these messages as if they were from the user, simply implement some MetaTransaction logic, maybe using a convenient library like Shipl, so that a user's signature can be submitted by your own key later on.

Closing this issue, as we've been clear that this proposed change is not safe for us to make, and so other approaches to the problems faced should be explored instead.