Calling `login` sometimes fails for a user until OpenMonero is restarted #154

skubakdj · 2019-06-08T15:20:28Z

I'm testing against a long-running stagenet instance of OpenMonero using the devel branch. At times, a user won't be able to call the login endpoint successfully and will be locked out until OpenMonero is restarted. Previously, the address/viewkey was able to log in just fine.

Here's the request:

curl  -w "\n" -X POST http://localhost:1984/login -d '{"address":"REDACTED","view_key":"REDACTED","create_account":true,"generated_locally":true,"withCredentials":false}'

The response:

{"reason":"Account creation failed","status":"error"}

Relevant lines from OM's log:

2019-06-08 14:48:19,698 [I,OM,OpenMoneroRequests.cpp:boost::optional<xmreg::XmrAccount> xmreg::OpenMoneroRequests::select_account(const string&, const string&, bool) const:2186] 5AUYJP: address does not exists
2019-06-08 14:48:19,700 [E,OM,OpenMoneroRequests.cpp:boost::optional<xmreg::XmrAccount> xmreg::OpenMoneroRequests::create_account(const string&, const string&, bool) const:2161] 5AUYJP: account creation failed

A restart of OpenMonero produces a successful response with the same request:

{"generated_locally":false,"new_address":false,"start_height":0,"status":"success"}

The text was updated successfully, but these errors were encountered:

moneroexamples · 2019-06-09T02:03:05Z

Previously, the address/viewkey was able to log in just fine.

Do you know roughly when previously? Like it started few days ago after my last commits?

moneroexamples · 2019-06-09T02:33:07Z

Also, when this happens, do you see any error in the backend printed out?

skubakdj · 2019-06-10T14:28:24Z

The time between the successful and failed requests was likely a few days. Currently running from devel commit cfdae8b5244e03efd458d01bccd6804fbe3ef825 (May 21st). I can update to the latest today.

The 'Relevant lines from OM's log' is what's printed out by the backend when the error occurs.

moneroexamples · 2019-06-10T21:44:24Z

The 'Relevant lines from OM's log' is what's printed out by the backend when the error occurs.

If something failed on the backend (e.g., mysql query, json parsing, etc), you should see some error about it on the backen'd console/log. The login issue is difficult to replicate, thus I don't know what could be the reason for that at present.

If you can't see anything in the logs, I can add more logging calls to login endpoint, to make it more verbose. This may help locating the issue faster?

skubakdj · 2019-06-11T20:14:23Z

Agree this is difficult to replicate, we haven't been able to reproduce the error since.

Unfortunately I wasn't able to find anything more helpful in the logs. More verbose logging for login would definitely be helpful 👍

skubakdj · 2019-06-11T20:48:08Z

Actually, this error has also been popping up in the logs:

[E,OM,CurrentBlockchainStatus.cpp:virtual void xmreg::CurrentBlockchainStatus::clean_search_thread_map():1156] Error in search thread: no_rows_inserted is zero!. It will be cleared.

I'm unable to confirm this is related, but just wanted to pass it along on the off-chance it might help debug anything.

#154 (comment)

moneroexamples · 2019-06-11T23:58:00Z

I added more details to logs bab01c8 . Hopefully this will allow pin down no_rows_inserted is zero and the login issues. For the no_rows_inserted is zero the logs should contain details of input and output for which it occurs. It could be that it fails, because there is already a duplicate record in mysql.

#154 (comment)

skubakdj · 2019-06-25T17:36:16Z

The same error cropped up again on devel with commit 405062c running on stagenet. Calls to login, get_address_txs, and get_address_info all returned errors. Here's the relevant log lines from OpenMonero, captured when calling get_address_txs:

[I,OM,CurrentBlockchainStatus.cpp:virtual bool xmreg::CurrentBlockchainStatus::start_tx_search_thread(xmreg::XmrAccount, std::unique_ptr<xmreg::TxSearch>):820] 5A8Fgb: TxSearch thread created.
[E,OM,TxSearch.cpp:virtual void xmreg::TxSearch::operator()():394] 5A8Fgb: insert outputs_found: no_rows_inserted is zero!Outputs:
{
    "account_id": 27,
    "amount": 7927917860000,
    "global_index": 1138515,
    "id": 0,
    "mixin": 11,
    "out_index": 1,
    "out_pub_key": "REDACTED",
    "timestamp": 1553009520,
    "tx_id": 2321,
    "tx_pub_key": "REDACTED"
}
[E,OM,TxSearch.cpp:virtual void xmreg::TxSearch::operator()():638] 5A8Fgb: exception in TxSearch: insert output_found: no_rows_inserted is zero!
[I,OM,TxSearch.cpp:virtual void xmreg::TxSearch::stop():661] 5A8Fgb: stopping the thread
[I,OM,TxSearch.cpp:virtual void xmreg::TxSearch::stop():661] 5A8Fgb: stopping the thread
[I,OM,CurrentBlockchainStatus.cpp:virtual void xmreg::CurrentBlockchainStatus::monitor_blockchain():56] Current blockchain height: 352322, pool size: 0 txs, no of TxSearch threads: 1
[E,OM,CurrentBlockchainStatus.cpp:virtual void xmreg::CurrentBlockchainStatus::clean_search_thread_map():1172] Error in search thread: insert output_found: no_rows_inserted is zero!. It will be cleared.
[I,OM,CurrentBlockchainStatus.cpp:virtual void xmreg::CurrentBlockchainStatus::clean_search_thread_map():1176] Ereasing a search thread
[I,OM,TxSearch.cpp:virtual xmreg::TxSearch::~TxSearch():667] 5A8Fgb: txSearch thread destroyed

Just like last time, restarting OpenMonero fixed things up.

skubakdj · 2019-06-25T17:47:50Z

I should also mention that we're experimenting with a setup that involves several instances of OpenMonero writing to the same database (in an attempt to scale things up). We've taken steps to prevent simultaneous search threads with the same address/viewkey and have had cursory success with multiple users connected to the system thus far.

Just wanted to provide some additional context in case that might factor into the cause.

moneroexamples · 2019-06-25T23:26:01Z

If you want to test how it performs under high load you can use this python script:

https://github.com/moneroexamples/openmonero/tree/master/scripts

You can use it to flood the backend with multiple requests.

skubakdj · 2019-07-01T16:59:25Z

I've been able to reproduce the error on a fresh install of OpenMonero and a clean (singleton, not shared) database. The user has agreed to provide their stagenet address and view key to this issue for troubleshooting. Here's some steps to reproduce:

Address - 5A8FgbMkmG2e3J41sBdjvjaBUyz8qHohsQcGtRf63qEUTMBvmA45fpp5pSacMdSg7A3b71RejLzB8EkGbfjp5PELVHCRUaE
View Key - 0f3fe25d0c6d4c94dde0c0bcc214b233e9c72927f813728b0f01f28f9d5e1201
Call login with create_account set to true
Call import_wallet_request
Periodically call get_address_info and get_address_txs

At block 283199 the error is produced and the user becomes locked out. Let me know if there's anything I can do to help troubleshoot this further!

moneroexamples · 2019-07-01T22:35:28Z

Will check. To clarify, this happens when you use the api calls directly, not the html frontend? If so, does it also happen when you use the frontend?

moneroexamples · 2019-07-01T22:39:42Z

Encountered this when importing your account in viewonly mode. Pasting this for future reference. Will look more into it.

# ERR: SQLException in /home/mwo2/openmonero/src/db/MySqlAccounts.cpp(insert) on line 334
# ERR: Duplicate entry '1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743' for key 'out_pub_key'
2019-07-01 22:31:10,498	E 5A8Fgb: insert outputs_found: no_rows_inserted is zero!Outputs: {"account_id":195,"amount":7927917860000,"global_index":1138515,"id":0,"mixin":11,"out_index":1,"out_pub_key":"1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743","timestamp":1552990505,"tx_id":12336,"tx_pub_key":"007caf7a553a894389dd562115b17e78ba84a5c7692677f216c54385dc5c6ff1"}


2019-07-01 22:31:10,514	E 5A8Fgb: exception in TxSearch: insert output_found: no_rows_inserted is zero!
2019-07-01 22:31:10,514	I 5A8Fgb: stopping the thread
2019-07-01 22:31:10,514	I 5A8Fgb: stopping the thread
2019-07-01 22:31:13,809	I Current blockchain height: 357388, pool size: 0 txs, no of TxSearch threads: 1
2019-07-01 22:31:13,809	E Error in search thread: insert output_found: no_rows_inserted is zero!. It will be cleared.
2019-07-01 22:31:13,809	I Ereasing a search thread
2019-07-01 22:31:13,809	I 5A8Fgb: txSearch thread destroyed
2019-07-01 22:31:18,995	I 5A8Fgb: TxSearch thread created.
# ERR: SQLException in /home/mwo2/openmonero/src/db/MySqlAccounts.cpp(insert) on line 334
# ERR: Duplicate entry '1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743' for key 'out_pub_key'
2019-07-01 22:31:19,107	E 5A8Fgb: insert outputs_found: no_rows_inserted is zero!Outputs: {"account_id":195,"amount":7927917860000,"global_index":1138515,"id":0,"mixin":11,"out_index":1,"out_pub_key":"1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743","timestamp":1552990505,"tx_id":12340,"tx_pub_key":"007caf7a553a894389dd562115b17e78ba84a5c7692677f216c54385dc5c6ff1"}


2019-07-01 22:31:19,113	E 5A8Fgb: exception in TxSearch: insert output_found: no_rows_inserted is zero!
2019-07-01 22:31:19,113	I 5A8Fgb: stopping the thread
2019-07-01 22:31:19,113	I 5A8Fgb: stopping the thread
2019-07-01 22:31:23,851	I Current blockchain height: 357389, pool size: 0 txs, no of TxSearch threads: 1
2019-07-01 22:31:23,851	E Error in search thread: insert output_found: no_rows_inserted is zero!. It will be cleared.
2019-07-01 22:31:23,851	I Ereasing a search thread
2019-07-01 22:31:23,851	I 5A8Fgb: txSearch thread destroyed
2019-07-01 22:31:28,986	I 5A8Fgb: TxSearch thread created.
# ERR: SQLException in /home/mwo2/openmonero/src/db/MySqlAccounts.cpp(insert) on line 334
# ERR: Duplicate entry '1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743' for key 'out_pub_key'
2019-07-01 22:31:29,095	E 5A8Fgb: insert outputs_found: no_rows_inserted is zero!Outputs: {"account_id":195,"amount":7927917860000,"global_index":1138515,"id":0,"mixin":11,"out_index":1,"out_pub_key":"1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743","timestamp":1552990505,"tx_id":12344,"tx_pub_key":"007caf7a553a894389dd562115b17e78ba84a5c7692677f216c54385dc5c6ff1"}


2019-07-01 22:31:29,100	E 5A8Fgb: exception in TxSearch: insert output_found: no_rows_inserted is zero!
2019-07-01 22:31:29,100	I 5A8Fgb: stopping the thread
2019-07-01 22:31:29,100	I 5A8Fgb: stopping the thread
2019-07-01 22:31:33,853	I Current blockchain height: 357389, pool size: 0 txs, no of TxSearch threads: 1
2019-07-01 22:31:33,853	E Error in search thread: insert output_found: no_rows_inserted is zero!. It will be cleared.
2019-07-01 22:31:33,853	I Ereasing a search thread
2019-07-01 22:31:33,853	I 5A8Fgb: txSearch thread destroyed
2019-07-01 22:31:38,989	I 5A8Fgb: TxSearch thread created.
# ERR: SQLException in /home/mwo2/openmonero/src/db/MySqlAccounts.cpp(insert) on line 334
# ERR: Duplicate entry '1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743' for key 'out_pub_key'
2019-07-01 22:31:39,100	E 5A8Fgb: insert outputs_found: no_rows_inserted is zero!Outputs: {"account_id":195,"amount":7927917860000,"global_index":1138515,"id":0,"mixin":11,"out_index":1,"out_pub_key":"1d1f059667d5984f0907693bec34d95698d373cbe42cb5178d3cc6973243b743","timestamp":1552990505,"tx_id":12348,"tx_pub_key":"007caf7a553a894389dd562115b17e78ba84a5c7692677f216c54385dc5c6ff1"}


2019-07-01 22:31:39,105	E 5A8Fgb: exception in TxSearch: insert output_found: no_rows_inserted is zero!
2019-07-01 22:31:39,105	I 5A8Fgb: stopping the thread
2019-07-01 22:31:39,105	I 5A8Fgb: stopping the thread
^C2019-07-01 22:31:40,775	I Request to finish the openmonero received
2019-07-01 22:31:40,775	I Stopping restbed service.
2019-07-01 22:31:40,775	I Stopping blockchain_monitoring_thread. Please wait.
2019-07-01 22:31:43,854	I 5A8Fgb: stopping the thread
2019-07-01 22:31:43,854	E Error in search thread: insert output_found: no_rows_inserted is zero!. It will be cleared.

skubakdj · 2019-07-01T23:01:33Z

Yep, I'm calling the API directly. I have not used the frontend extensively, so I'm not sure if it also pops up there.

moneroexamples · 2019-07-02T12:55:21Z

The problem is caused by two different txs having same outputs sent to your address:

http://139.162.60.17:8082/search?value=508cca34157aab8399a09c46e999ed2191f2473f46dafbdbd6fc36d7299a9100

http://139.162.60.17:8082/search?value=54f6b3a51baaac0306e9f299fd630cbde6e5255b419af4d35605a83da12e4485

Thus TxSearch terminates when processing second tx as it is assumed that outputs are unique. OM does not allow for such a situation at present. Need some time to fix it, as it requires changes to db. The uniqueness of outputs is enforced at sql level:

openmonero/sql/openmonero.sql

Line 94 in 507bda0

UNIQUE KEY `out_pub_key` (`out_pub_key`),

moneroexamples · 2019-07-03T10:01:44Z

Would you mind providing the seed for the stagenet wallet in question?

skubakdj · 2019-07-05T17:22:32Z

Here's the seed with the key details:

      ---------------------------------------------------------------------------------------------------
      | Monero Electrum words :  tavern judge beyond bifocals deepest mural onward dummy                |
      |                          eagle diode gained vacation rally cause firm idled                     |
      |                          jerseys moat vigilant upload bobsled jobs cunning doing jobs           |
      |                                                                                                 |
      | seed                  :  db9e57474be8b64118b6acf6ecebd13f8f7c326b3bc1b19f4546573d6bac9dcf       |
      | Spend key             :  3b094ca7218f175e91fa2402b4ae239a2fe8262792a3e718533a1a357a1e4109       |
      | View key              :  0f3fe25d0c6d4c94dde0c0bcc214b233e9c72927f813728b0f01f28f9d5e1201       |
      ---------------------------------------------------------------------------------------------------

moneroexamples · 2019-07-09T11:33:51Z

You can check the following branch:

https://github.com/moneroexamples/openmonero/tree/fix_nonunique_outputs

It allows for non-unique output public keys. However, txs that you are doing (those with non-unique keys) are highly unusual and not even handled correctly by official monero wallet from what I can see. Thus its very difficult for me to test the changes.

I don't know yet if this branch will get merged to OM, as I'm inclined to say that your tx creation procedure that you are using is incorrect. Thus maybe its for the better that such txs cause OM to fail, so the issue could be detected.

cslashm · 2019-07-17T08:12:43Z

@moneroexamples thanks for having investigated this issue. This test wallet is mine, the one I use for developing Ledger Nano S/X Integration.

So those "non std" TX have certainly been generated during my development and test. I know that when I full resync this wallet with monero-cli I have some WARN/ERR about few old TX.

Maybe a simple solution is to discard such TX with a warning. I don't know if it is acceptable, but as such TX should not exist it could the simplest solution.

moneroexamples · 2019-07-17T10:33:24Z

Maybe a simple solution is to discard such TX with a warning. I don't know if it is acceptable, but as such TX should not exist it could the simplest solution.

I think is is good middle-ground. So I will check how can I handle that.

#154 (comment)

moneroexamples · 2019-07-18T10:45:55Z

You can check devel branch. Instead of throwing an exception and terminating search threads, an error message is displayed. On the error, current the threads moves on to next tx.

skubakdj · 2019-07-26T13:22:16Z

I can confirm that the latest in devel now allows the account to import without issue.

Thanks so much @moneroexamples and @cslashm!

moneroexamples added a commit that referenced this issue Jun 11, 2019

Add more detailes to logs

bab01c8

#154 (comment)

moneroexamples added a commit that referenced this issue Jun 12, 2019

Add more detailes to logs

405062c

#154 (comment)

moneroexamples added a commit that referenced this issue Jul 18, 2019

dont throw on output and key images failure to insert

1e6c13f

#154 (comment)

skubakdj closed this as completed Jul 26, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Calling `login` sometimes fails for a user until OpenMonero is restarted #154

Calling `login` sometimes fails for a user until OpenMonero is restarted #154

skubakdj commented Jun 8, 2019

moneroexamples commented Jun 9, 2019

moneroexamples commented Jun 9, 2019

skubakdj commented Jun 10, 2019

moneroexamples commented Jun 10, 2019

skubakdj commented Jun 11, 2019

skubakdj commented Jun 11, 2019

moneroexamples commented Jun 11, 2019

skubakdj commented Jun 25, 2019

skubakdj commented Jun 25, 2019

moneroexamples commented Jun 25, 2019

skubakdj commented Jul 1, 2019

moneroexamples commented Jul 1, 2019

moneroexamples commented Jul 1, 2019

skubakdj commented Jul 1, 2019

moneroexamples commented Jul 2, 2019 •

edited

moneroexamples commented Jul 3, 2019

skubakdj commented Jul 5, 2019

moneroexamples commented Jul 9, 2019

cslashm commented Jul 17, 2019

moneroexamples commented Jul 17, 2019

moneroexamples commented Jul 18, 2019

skubakdj commented Jul 26, 2019

Calling login sometimes fails for a user until OpenMonero is restarted #154

Calling login sometimes fails for a user until OpenMonero is restarted #154

Comments

skubakdj commented Jun 8, 2019

moneroexamples commented Jun 9, 2019

moneroexamples commented Jun 9, 2019

skubakdj commented Jun 10, 2019

moneroexamples commented Jun 10, 2019

skubakdj commented Jun 11, 2019

skubakdj commented Jun 11, 2019

moneroexamples commented Jun 11, 2019

skubakdj commented Jun 25, 2019

skubakdj commented Jun 25, 2019

moneroexamples commented Jun 25, 2019

skubakdj commented Jul 1, 2019

moneroexamples commented Jul 1, 2019

moneroexamples commented Jul 1, 2019

skubakdj commented Jul 1, 2019

moneroexamples commented Jul 2, 2019 • edited

moneroexamples commented Jul 3, 2019

skubakdj commented Jul 5, 2019

moneroexamples commented Jul 9, 2019

cslashm commented Jul 17, 2019

moneroexamples commented Jul 17, 2019

moneroexamples commented Jul 18, 2019

skubakdj commented Jul 26, 2019

Calling `login` sometimes fails for a user until OpenMonero is restarted #154

Calling `login` sometimes fails for a user until OpenMonero is restarted #154

moneroexamples commented Jul 2, 2019 •

edited