Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Moderate load on local replica (or emulator) lead to canister calls hanging indefinitely #1574

Open
megrogan opened this issue Apr 7, 2021 · 6 comments
Open

Moderate load on local replica (or emulator) lead to canister calls hanging indefinitely #1574

megrogan opened this issue Apr 7, 2021 · 6 comments
Assignees
@ericswanson-dfinity
Labels
blocked bug Something isn't working triaged

Comments

@megrogan
Copy link

megrogan commented Apr 7, 2021 •
edited

Steps to reproduce:

  1. Install dfx 0.7.0-beta.2
    DFX_VERSION=0.7.0-beta.2 sh -ci "$(curl -fsSL https://sdk.dfinity.org/install.sh)"
  2. Start dfx (issue happens much more quickly when running emulator)
    dxf start --emulator
  3. Clone "open chat"
    open-chat-labs/open-chat@04162d2
  4. Build and deploy "open chat" locally
    3.1 cd into open-chat
    3.2 npm install
    3.3 dfx deploy
  5. Open local "website" canister in browser, sign-in, and register a user name
  6. Repeat step 4 in another browser - say Chrome incognito
  7. Open chrome dev tools and watch the network tab (open chat polls for new messages)
  8. After a few minutes you should see the query calls hanging
@megrogan
Copy link
Author

megrogan commented Apr 8, 2021

Btw, the same issue occurs with just one instance of "open chat" it just takes longer

@p-shahi p-shahi removed their assignment Apr 19, 2021
@p-shahi p-shahi added the bug Something isn't working label Apr 19, 2021
@p-shahi
Copy link
Contributor

p-shahi commented Apr 19, 2021

Has this been debugged and resolved? We noticed that a lot last week when working on the identity provider, and it almost derailed our Dom demo…
(My gut feeling debugging seems to point that some component in dfx runs out of something like connections, as if it doesn’t know that it has closed connections and eventually doesn’t allow new ones.)

@nomeata

@ericswanson-dfinity ericswanson-dfinity self-assigned this Apr 26, 2021
@taylorham
Copy link
Contributor

taylorham commented Apr 27, 2021 •
edited

Affecting CanCan as well. Seems to consistently happen to us after around 9 minutes of the replica being started, but that seems more of a coincidence than a cause.

@hansl has suggested this could also be related to a hang we're experiencing during the asset upload phase of dfx deploy/dfx canister install..., where the replica becomes completely unresponsive after uploading a portion of assets and we have to cancel the process. dfx stop then prints the "Stopping the replica... Stopped." output, but the dfx process is never stopped and has to be manually killed. Let me know if I should create another ticket for this one, @ericswanson-dfinity!

@nomeata
Copy link
Contributor

nomeata commented Apr 27, 2021

I worked around it locally by not using dfx start but rather running icx-proxy; instructions in https://github.com/dfinity/idp-service/blob/main/Demo.md. This may be useful for people who need this working right now more reliably for a Demo.

@ericswanson-dfinity
Copy link
Member

Trying to repro, I get stuck at step

  1. Open local "website" canister in browser, sign-in, and register a user name

Screen Shot 2021-04-26 at 3 35 18 PM

@ericswanson-dfinity ericswanson-dfinity mentioned this issue Apr 27, 2021
Closed
@sesi200 sesi200 added needs-triage This ticket needs investigation. triaged and removed needs-triage This ticket needs investigation. labels Oct 5, 2022
@sesi200
Copy link
Contributor

sesi200 commented Oct 6, 2022

This could also be releated to the issues mentioned by Mio here

dfinity-bot added a commit that referenced this issue Feb 8, 2023
@dfinity-bot
build: niv advisory-db: update bb92d2d5 -> b485cf4d
416d4d2 
## Changelog for advisory-db:
Branch: main
Commits: [rustsec/advisory-db@bb92d2d5...b485cf4d](rustsec/advisory-db@bb92d2d...b485cf4)

* [`58e20bb7`](rustsec/advisory-db@58e20bb) Add CVE-2023-0286 for openssl-src ([RustSec/advisory-db⁠#1573](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1573))
* [`56f79c7b`](rustsec/advisory-db@56f79c7) Assigned RUSTSEC-2023-0006 to openssl-src ([RustSec/advisory-db⁠#1574](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1574))
* [`a0530f10`](rustsec/advisory-db@a0530f1) Add CVE-2022-4304 for openssl-src ([RustSec/advisory-db⁠#1575](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1575))
* [`d70e7f95`](rustsec/advisory-db@d70e7f9) Assigned RUSTSEC-2023-0007 to openssl-src ([RustSec/advisory-db⁠#1576](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1576))
* [`4bf8b4c3`](rustsec/advisory-db@4bf8b4c) Add CVE-2022-4203 for openssl-src ([RustSec/advisory-db⁠#1577](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1577))
* [`8d1c5992`](rustsec/advisory-db@8d1c599) Assigned RUSTSEC-2023-0008 to openssl-src ([RustSec/advisory-db⁠#1578](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1578))
* [`818cdec0`](rustsec/advisory-db@818cdec) Add CVE-2023-0215 for openssl-src ([RustSec/advisory-db⁠#1579](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1579))
* [`b324cf92`](rustsec/advisory-db@b324cf9) Assigned RUSTSEC-2023-0009 to openssl-src ([RustSec/advisory-db⁠#1581](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1581))
* [`1445e637`](rustsec/advisory-db@1445e63) Add CVE-2022-4450 for openssl-src ([RustSec/advisory-db⁠#1580](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1580))
* [`94fdfaf7`](rustsec/advisory-db@94fdfaf) Assigned RUSTSEC-2023-0010 to openssl-src ([RustSec/advisory-db⁠#1585](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1585))
* [`3e3631fd`](rustsec/advisory-db@3e3631f) Add CVE-2023-0216 for openssl-src ([RustSec/advisory-db⁠#1582](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1582))
* [`19c2e8af`](rustsec/advisory-db@19c2e8a) Assigned RUSTSEC-2023-0011 to openssl-src ([RustSec/advisory-db⁠#1586](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1586))
* [`3cfb8970`](rustsec/advisory-db@3cfb897) Add CVE-2023-0217 for openssl-src ([RustSec/advisory-db⁠#1583](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1583))
* [`daa52cad`](rustsec/advisory-db@daa52ca) Assigned RUSTSEC-2023-0012 to openssl-src ([RustSec/advisory-db⁠#1587](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1587))
* [`0f3d1597`](rustsec/advisory-db@0f3d159) Add CVE-2023-0401 for openssl-src ([RustSec/advisory-db⁠#1584](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1584))
* [`feb5cddd`](rustsec/advisory-db@feb5cdd) Assigned RUSTSEC-2023-0013 to openssl-src ([RustSec/advisory-db⁠#1588](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1588))
* [`c6193701`](rustsec/advisory-db@c619370) Add advisory aliyun-oss-client ([RustSec/advisory-db⁠#1589](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1589))
* [`8a1400e3`](rustsec/advisory-db@8a1400e) Assigned RUSTSEC-2022-0089 to aliyun-oss-client ([RustSec/advisory-db⁠#1590](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1590))
* [`cb6cfde9`](rustsec/advisory-db@cb6cfde) Update RUSTSEC-2020-0071.md ([RustSec/advisory-db⁠#1591](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1591))
* [`b485cf4d`](rustsec/advisory-db@b485cf4) Update RUSTSEC-2020-0071.md ([RustSec/advisory-db⁠#1594](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1594))
mergify bot pushed a commit that referenced this issue Mar 4, 2023
@dfinity-bot
build: niv advisory-db: update bb92d2d5 -> b485cf4d (#2956)
4461fd3 
## Changelog for advisory-db:
Branch: main
Commits: [rustsec/advisory-db@bb92d2d5...b485cf4d](rustsec/advisory-db@bb92d2d...b485cf4)

* [`58e20bb7`](rustsec/advisory-db@58e20bb) Add CVE-2023-0286 for openssl-src ([RustSec/advisory-db⁠#1573](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1573))
* [`56f79c7b`](rustsec/advisory-db@56f79c7) Assigned RUSTSEC-2023-0006 to openssl-src ([RustSec/advisory-db⁠#1574](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1574))
* [`a0530f10`](rustsec/advisory-db@a0530f1) Add CVE-2022-4304 for openssl-src ([RustSec/advisory-db⁠#1575](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1575))
* [`d70e7f95`](rustsec/advisory-db@d70e7f9) Assigned RUSTSEC-2023-0007 to openssl-src ([RustSec/advisory-db⁠#1576](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1576))
* [`4bf8b4c3`](rustsec/advisory-db@4bf8b4c) Add CVE-2022-4203 for openssl-src ([RustSec/advisory-db⁠#1577](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1577))
* [`8d1c5992`](rustsec/advisory-db@8d1c599) Assigned RUSTSEC-2023-0008 to openssl-src ([RustSec/advisory-db⁠#1578](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1578))
* [`818cdec0`](rustsec/advisory-db@818cdec) Add CVE-2023-0215 for openssl-src ([RustSec/advisory-db⁠#1579](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1579))
* [`b324cf92`](rustsec/advisory-db@b324cf9) Assigned RUSTSEC-2023-0009 to openssl-src ([RustSec/advisory-db⁠#1581](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1581))
* [`1445e637`](rustsec/advisory-db@1445e63) Add CVE-2022-4450 for openssl-src ([RustSec/advisory-db⁠#1580](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1580))
* [`94fdfaf7`](rustsec/advisory-db@94fdfaf) Assigned RUSTSEC-2023-0010 to openssl-src ([RustSec/advisory-db⁠#1585](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1585))
* [`3e3631fd`](rustsec/advisory-db@3e3631f) Add CVE-2023-0216 for openssl-src ([RustSec/advisory-db⁠#1582](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1582))
* [`19c2e8af`](rustsec/advisory-db@19c2e8a) Assigned RUSTSEC-2023-0011 to openssl-src ([RustSec/advisory-db⁠#1586](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1586))
* [`3cfb8970`](rustsec/advisory-db@3cfb897) Add CVE-2023-0217 for openssl-src ([RustSec/advisory-db⁠#1583](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1583))
* [`daa52cad`](rustsec/advisory-db@daa52ca) Assigned RUSTSEC-2023-0012 to openssl-src ([RustSec/advisory-db⁠#1587](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1587))
* [`0f3d1597`](rustsec/advisory-db@0f3d159) Add CVE-2023-0401 for openssl-src ([RustSec/advisory-db⁠#1584](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1584))
* [`feb5cddd`](rustsec/advisory-db@feb5cdd) Assigned RUSTSEC-2023-0013 to openssl-src ([RustSec/advisory-db⁠#1588](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1588))
* [`c6193701`](rustsec/advisory-db@c619370) Add advisory aliyun-oss-client ([RustSec/advisory-db⁠#1589](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1589))
* [`8a1400e3`](rustsec/advisory-db@8a1400e) Assigned RUSTSEC-2022-0089 to aliyun-oss-client ([RustSec/advisory-db⁠#1590](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1590))
* [`cb6cfde9`](rustsec/advisory-db@cb6cfde) Update RUSTSEC-2020-0071.md ([RustSec/advisory-db⁠#1591](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1591))
* [`b485cf4d`](rustsec/advisory-db@b485cf4) Update RUSTSEC-2020-0071.md ([RustSec/advisory-db⁠#1594](http://r.duckduckgo.com/l/?uddg=https://github.com/RustSec/advisory-db/issues/1594))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ericswanson-dfinity ericswanson-dfinity

Labels
blocked bug Something isn't working triaged
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@nomeata @megrogan @taylorham @sesi200 @p-shahi @ericswanson-dfinity