New version of bridge fetches huge amount of data

[frozencemetery]

When launching proton-bridge at 27e7d79, it now fetches a huge amount of data on start - logs suggest it's downloading every message and storing it on disk.

Expected Behavior

I'm already using bridge to download messages to this machine - there's no need for bridge to store an additional copy.

Current Behavior

Since what I'm guessing is 6bd0739, hundreds of megs of encrypted blobs are fetched and stored in ~/.config/protonmail

Possible Solution

If, as it seems, this is intentional, maybe an option to turn it off? Otherwise it seems like this is going to eat as much disk as one has quota in protonmail.

Steps to Reproduce

1. Have decent sized mailbox and run bridge at b0ce46c.
2. Upgrade bridge to 27e7d79

Version Information

27e7d79 (looks like this will become 2.0.1). It presumably happened earlier, but it's hard to test.

Context (Environment)

Linux, built from git, running headless. Email is pulled over IMAP for use with another client. This is a single machine setup, so the other client stores a copy and so does bridge.

Detailed Description

I'd like bridge to not store all this on disk, but as it seems intentional, perhaps an option to turn this behavior off?

Possible Implementation

I don't write Go and am not very familiar with the tree, sorry.

[andrzejsza]

yes, this is very much by design - Bridge now stores encrypted message bodies to be able to serve them to the client faster. yes, you can switch this off - go to Settings/Advanced settings/Local cache - you can disable it there.

[frozencemetery]

Ah, and I see it's cache_enabled in ~/.config/protonmail/bridge/prefs.json (I run headless). Thanks!

[tomtastic]

This does seem to be an issue (MacOS 12, Bridge 2.1.1), I was just doing an initial setup/sync of my ~600MB account and Bridge has downloaded over 20GB since it began.

Here we can see it's downloaded just over 2GB since I restarted the process :

$ nettop -t wifi -p $(pgrep -f 'ProtonMail Bridge')

In a log file spanning just one hour, there are GETs for the same items repeated many times over and over :

$ head -1 v2.1.1_61e4ca5814_1646314262.log; tail -1 v2.1.1_61e4ca5814_1646314262.log
INFO[Mar  3 13:31:02.719] No need to migrate file, source doesn't exist  destination="/Users/tmatoo/Library/Application Support/protonmail/bridge/prefs.json" source=/Users/tmatoo/Library/Caches/protonmail/bridge/c11/prefs.json
INFO[Mar  3 14:31:09.390] Requesting GET /mail/v4/attachments/dw-UHZojI5BIqjxJuJ765IzWqh4etrjWgBWarWVU5YGPqAfKMZmxlcFKAqZoICv9Fm1sv7AX1N81r9XUM7eGdg==  pkg=pmapi

$ grep '/mail/v4/messages' v2.1.1_61e4ca5814_1646314262.log | cut -d" " -f5- | sort | uniq -c | sort -n | tail -10
  37 Requesting GET /mail/v4/messages/33i_tN3S6M0cAgxxjWwHqsK8hPGmIhZ3YByarxJl4X5l5K96e3AVycC81QaU-Z9uzP8K856CF0AX3VZngVpl_Q==  pkg=pmapi
  40 Requesting GET /mail/v4/messages/k3HIAdUXDbbkn0nFEgWVNoBuN9zPJSBQBriiff9M3iFKO8hsSvxCBcc2cYp7o3Bgl2rLyUmfhcXu9LuK7QJkIA==  pkg=pmapi
  41 Requesting GET /mail/v4/messages/IqR8tKcOW8DU_dB-HguNanWttHj5MJ-uj4VfrueY7LqQnHvk4duQJN9UusIRiRt9d55kmqOEsLhofJx4ZbyU6Q==  pkg=pmapi
  41 Requesting GET /mail/v4/messages/n7XLqz1ciVDyfcW7qjG-J45wPKa6oPRKkTeFvBaqsgKITzdJ7ifcsU28NtDezM6sVqR5D3YZ9hWr16jMv6_dvQ==  pkg=pmapi
  43 Requesting GET /mail/v4/messages/VB9kpQD7trC1NJNFQOqJrKsGS7vQJ_uMVLAIdM7HUQ3HkuTuvLDn9_aKzEfDSh7SJ7BSpE2TO1m6gPgTguOcRg==  pkg=pmapi
  52 Requesting GET /mail/v4/messages/LMJNYAxL2wvdpPfKXoNA8jEl7aMII29aAnR3g6qsbW2HweLymecEClKOdyNTvAi1CSu3pK-CQCkZn4H1FoDj5w==  pkg=pmapi
  55 Requesting GET /mail/v4/messages/mTVSNVjo09Lw98AvJOozJxzgGOVG4dNCdrfd1rkxK9RQCrqzM0CMGHEUBf7IptAMvryulxvKvCAb04QprEdybQ==  pkg=pmapi
  56 Requesting GET /mail/v4/messages/ClEc8jIP6sZ_3WTg9ZfRlgKcLK7856q1C6wuC5mFgMBzA77UC2ypFhCO8md17BpC3KZRJt0dAFc7lyd-rUQcLQ==  pkg=pmapi
  72 Requesting GET /mail/v4/messages/XIxa_RwCM5TtxwirCfLiPS3EA7V3QSEm7x0vmO5zfck-ksf72yWXvnPD97uPJZ6pbPYNIUmMQ8G6LSuD5jHEug==  pkg=pmapi
  72 Requesting GET /mail/v4/messages/bLZrGJdt92eqo2EgiskyvnWrqhc9Y0bm0Vnywp2zPr1FR_Hd7y_5PcJO7brffyEGqjwE5z_8e-SL_MqWhrwkUA==  pkg=pmapi

$ grep '/mail/v4/att' v2.1.1_61e4ca5814_1646314262.log | cut -d" " -f5- | sort | uniq -c | sort -n | tail -10
  36 Requesting GET /mail/v4/attachments/zahuHHiiNXuIhhJfbO97lNe9onDOHI0PLZzzkF5fX3LEYz3VKUyncZumMrFT3lDlrqIKKVfmBoKgHwNhqcsO4g==  pkg=pmapi
  41 Requesting GET /mail/v4/attachments/26D4uTDAnnoDyqrT1qKLF_weKpsmaJNkWyR1FHaS1rXflSS0Vly8GNKyZxW1UzxYoUZeuF5Bv6hAXBWzDMS0_g==  pkg=pmapi
  41 Requesting GET /mail/v4/attachments/sIwp3vxj61OZQkm3s6a-XFbmUilxjKT_CABIyCkcRVSD-2ZejuSjw9VzRegpurkvgZpOGC0uXa-ziv3vZuUE4g==  pkg=pmapi
  41 Requesting GET /mail/v4/attachments/vxF5ZV1gmLaA_akAsuURstF5w2KqS7mvQ72fr9UHz-HDbExXEBSiU_T7O0sfPuTxdXU5GqgwuhPR9WbqnG4hJw==  pkg=pmapi
  43 Requesting GET /mail/v4/attachments/fiRCAup_bRiM4I9oMfDxOmx4CveGqEz2LrC99EKIdxyzj3zmNzOUiTwD50tyuiWJM9cXrV10LdZaJWFZKmcx4A==  pkg=pmapi
  52 Requesting GET /mail/v4/attachments/0OVEoXKFhvxilFcQJ0L1_Fp6k7qXAv_p6jw_XowzvCqiYkoRn9ha_nTt0ekOfQUA7Tcd-Cyn4tKBoerexDlvHg==  pkg=pmapi
  52 Requesting GET /mail/v4/attachments/pMaHBpDdPvVcsXndcrvt6kJCC9PG2Tw8plnmV_kWxN2xKkkm8PkfFY718gyuaviDA2H1XiCo0frP6ZUkQviAzQ==  pkg=pmapi
  55 Requesting GET /mail/v4/attachments/2zJtOg1GQqnlVtk6uzP1iXSboT26k448FSPm9lJD4L9-Fmuz-4cUaIrSlS-xEeROWQviMUnIPtPkzGLf7sSNzw==  pkg=pmapi
  56 Requesting GET /mail/v4/attachments/Yx807JKUTwHbq2eBDLcRskt74cNh3KN5ZY7RZm8PsbTG7N1_pys34D3Ytlh8yr1RAU1lRX3H6otiX1519pN5gA==  pkg=pmapi
  56 Requesting GET /mail/v4/attachments/nEfSEuybuM_ggsMtNDORMe8isZ43KZff-pVulo0YYDuv6qS2vgmAii8Xw7ErVQhQtPoRMZk9YIQI3b9bZ6r8Aw==  pkg=pmapi

$ grep '/event' v2.1.1_61e4ca5814_1646314262.log | cut -d" " -f5- | sort | uniq -c | sort -n | tail -10
   3 Requesting GET /events/1x7X63N7cdc86WjpUqOAXEb3e-1xhWFhG-CsGLrEOTJvfkAzmMzK1XyBl_oFFBy0trxq0RaokeKKrsS-oIZY5Q==  pkg=pmapi
   3 Requesting GET /events/Ejht13RyYLx2n4PypD7zVRWv17q7fX4kY5rokt7XMErqXgvIyeBj8KSUBLsNuJFACH1sW2Eu1AkGGSdMYfT0dA==  pkg=pmapi
   3 Requesting GET /events/bMpRNA8pfOTkBUdx7pp84mSQ8SOmBUerC8wLACb10tq5pZMAKUp_2e1y-RnTyccCo6jLSdV_VcQELRe2iRF3Rg==  pkg=pmapi
   6 Requesting GET /events/dkTkMA5uUC5GKeQzv1N9g0Luorj7MDX1L2-7g8TTIlZXoiGsABaiVtSWfr0HWyRgZQLagZoExcB2ajIuX_nE5w==  pkg=pmapi
   7 Requesting GET /events/2Z6YuC5mUWp5KZMZX1yOgb-VYKCulNNxLPuCMkvFIlRrujNB9ZK4_XZn-4Sn0Lbt-01pN16eB_TnRmHUxf4A4g==  pkg=pmapi
   7 Requesting GET /events/hPwCepv2-9SBlYvI0gXLL_6gks0kZvU8SLdGYdmF_1VCX6GdaVtxeNr-F_mTrJ8fc5vUrrf4pSpY8p_lacUCZQ==  pkg=pmapi
  10 Requesting GET /events/-b0veOcyNBQcPdgRQFGYm0F2Qdh9yq_wD2IkKVcpSCDJtL_Vh3s_bX6XPBD9Ku_ahRI_uPjm9o6kxPosBChavg==  pkg=pmapi
  15 Requesting GET /events/1xiSE__jM6Jh8TbtNbhWbkadc2Qj_1IY6gV8b2amuPYvZJCl8Em-Z_E7zsPWJ8Vfw6L0uHf7KqbJ9Caz23u40w==  pkg=pmapi
  25 Requesting GET /events/38fOo71HqLhLVBc6HQtq8bpfcK1ytW0ooXkb40u5JVFrwGBFf-fjxquXvlJfYiL_yNuBmg-SMLeU5ODFcxP-SQ==  pkg=pmapi
  52 Requesting GET /events/OXnYvhPE2OSTOzeS2h47eK0f1fLrZvSngpulD0DQLzt-ontzx5phlmSCOk3n03UfNiE0E-oTtuc_7PtZZS4xSw==  pkg=pmapi

[tomtastic]

I tried disabling the cache and re-syncing from scratch, but it didn't make any difference, still ridiculous amounts of data downloaded and Apple Mail was struggling to send even a single email, and selecting an un-synced email would often simply timeout.

I've now reverted to Bridge v1.8.12 where this behaviour doesn't seem to exist (almost fully synced and just 1.2GB of data in total, vs the 20GB+ previously), and the end user experience with Apple Mail is far more responsive.

Checking the logs over a similar hour long period of syncing as before, indicates that v1.8.12 is requesting a GETs of unique items an order of magnitude less often.

$ head -1 v1.8.12_b0ce46ca8a_1646408253.log; tail -1 v1.8.12_b0ce46ca8a_1646408253.log
INFO[Mar  4 15:37:33.781] No need to migrate file, source doesn't exist  destination="/Users/tmatoo/Library/Application Support/protonmail/bridge/prefs.json" source=/Users/tmatoo/Library/Caches/protonmail/bridge/c11/prefs.json
INFO[Mar  4 16:39:13.850] Requesting GET /mail/v4/messages/Vn8lAo54CkVv4X5CDa44OY25WgCENNB1yFTZIEwuB5_wHa9z7oQhUtKM2wNq3EHFb77rDHHsgzf4Rb36q1ruiA==  pkg=pmapi

$ grep '/mail/v4/messages' v1.8.12_b0ce46ca8a_1646408253.log| cut -d" " -f5- | sort | uniq -c | sort -n | tail -10
   3 Requesting GET /mail/v4/messages/vG_cmx-uACM2vpSMVxUo0WmQvhaNMxDopPuNr3ek2bNAnAV4VSzaLNBMuxJak3M6dTNTytLTR4muQn2-kv6tAA==  pkg=pmapi
   3 Requesting GET /mail/v4/messages/wtcXFIRWdYbb9ZqYVuWQXDHqwmAGjOWwvdLsprLjXIQ7peO4_3fbHkjMtJiQ9pWL0w9uMGX8Ljp36LeJ2c3l6A==  pkg=pmapi
   4 Requesting GET /mail/v4/messages/9SWYPYzSQ6sZHG4gBcegGOz3FOjMyB6VSG4Y5TMEZMb935gsA5GiirdZ9mWGI-9yYEYGHPkU0afPKYx5W7OSsA==  pkg=pmapi
   4 Requesting GET /mail/v4/messages/IqR8tKcOW8DU_dB-HguNanWttHj5MJ-uj4VfrueY7LqQnHvk4duQJN9UusIRiRt9d55kmqOEsLhofJx4ZbyU6Q==  pkg=pmapi
   4 Requesting GET /mail/v4/messages/k3HIAdUXDbbkn0nFEgWVNoBuN9zPJSBQBriiff9M3iFKO8hsSvxCBcc2cYp7o3Bgl2rLyUmfhcXu9LuK7QJkIA==  pkg=pmapi
   4 Requesting GET /mail/v4/messages/n7XLqz1ciVDyfcW7qjG-J45wPKa6oPRKkTeFvBaqsgKITzdJ7ifcsU28NtDezM6sVqR5D3YZ9hWr16jMv6_dvQ==  pkg=pmapi
   4 Requesting GET /mail/v4/messages/txiE6RX0T4Dx9eUe_yeBdkpw9hAFTXcjof4EnLNU_BIzN__opQTdjG7sz-YwOwzu-ShqqKo9fN5wAgJUZgnMfA==  pkg=pmapi
   4 Requesting GET /mail/v4/messages/yr07-82RtELnYeKq8urjTs4L-xqsOtUDMh7_VBD6wR_l4ZEKp08czfkIjinxSvBYZGi4vlPBXbR1GrkvA2R4PQ==  pkg=pmapi
   5 Requesting GET /mail/v4/messages/nrbniIHEjJic3b-xsIv3D42Jx7M5kv7FKmJLwgyN-oh4I1RCf2GRw0EMr010dzn4n7BGboN0jUdfO_9CI0dN0A==  pkg=pmapi
  30 Requesting GET /mail/v4/messages              pkg=pmapi

$ grep '/mail/v4/att' v1.8.12_b0ce46ca8a_1646408253.log| cut -d" " -f5- | sort | uniq -c | sort -n | tail -10
   3 Requesting GET /mail/v4/attachments/sFrJLgpRJtcmGxQgckPKdlgPWw7T2jDaBxyanMPJvXOzctVzOqjAvqmo8dORxalrewRczb3egX9rz5WoIH1sFw==  pkg=pmapi
   3 Requesting GET /mail/v4/attachments/uK6agRJkbustf8KvP-I-fJTgu7vmXDlNKv4F4W0ju9UqpAt9gciZ9DAEdAodN5Hmc2uspsgjiemRB2OSmAG6Kw==  pkg=pmapi
   3 Requesting GET /mail/v4/attachments/zahuHHiiNXuIhhJfbO97lNe9onDOHI0PLZzzkF5fX3LEYz3VKUyncZumMrFT3lDlrqIKKVfmBoKgHwNhqcsO4g==  pkg=pmapi
   4 Requesting GET /mail/v4/attachments/26D4uTDAnnoDyqrT1qKLF_weKpsmaJNkWyR1FHaS1rXflSS0Vly8GNKyZxW1UzxYoUZeuF5Bv6hAXBWzDMS0_g==  pkg=pmapi
   4 Requesting GET /mail/v4/attachments/9um1UE3iRELdkVBDXWqliBrdeKs03oSS_XNOU1hS52gnUfUq-_v7JxfHLztOxPR_bPQjLFBhGS31Rfu1NntPow==  pkg=pmapi
   4 Requesting GET /mail/v4/attachments/O0DXF6Ub69XVTmewDV511F2KveFk_Nhu832Dxhqk9344jFBBBot2SCy3JyMtwtqmSVCztQAXreBeg1Bq3LRV_g==  pkg=pmapi
   4 Requesting GET /mail/v4/attachments/dw-UHZojI5BIqjxJuJ765IzWqh4etrjWgBWarWVU5YGPqAfKMZmxlcFKAqZoICv9Fm1sv7AX1N81r9XUM7eGdg==  pkg=pmapi
   4 Requesting GET /mail/v4/attachments/p5vwSJbrxWbPIdYV-wJg2bZ-wdUa_oL2eYImwg8d4J_Clg3SQXqJLXh56Fa1KbEGftOCLhapqGxb568k27B8TA==  pkg=pmapi
   4 Requesting GET /mail/v4/attachments/sIwp3vxj61OZQkm3s6a-XFbmUilxjKT_CABIyCkcRVSD-2ZejuSjw9VzRegpurkvgZpOGC0uXa-ziv3vZuUE4g==  pkg=pmapi
   4 Requesting GET /mail/v4/attachments/vxF5ZV1gmLaA_akAsuURstF5w2KqS7mvQ72fr9UHz-HDbExXEBSiU_T7O0sfPuTxdXU5GqgwuhPR9WbqnG4hJw==  pkg=pmapi

$ grep '/event' v1.8.12_b0ce46ca8a_1646408253.log| cut -d" " -f5- | sort | uniq -c | sort -n | tail -10
   1 Requesting GET /events/latest                 pkg=pmapi
   2 Requesting GET /events/IOpbr1v8OODM_JRS9tgB-H00qfo5K0vE28KFh19tUXWR1zYgT10MM57TcYrJ8NP3qk-ZpGgni3uNnhjD1GFq6w==  pkg=pmapi
   2 Requesting GET /events/ZP1DrCs9rJMOzrSJz1dbrbhkvSzNomBXF3N67YHs2bzz4IYiZXZNwF5AJ4VVkNSUT1PJLVHTIFE1NaKcwVOVwQ==  pkg=pmapi
   6 Requesting GET /events/LEEkbBu__byiCe1LgH6jg8erwORJZh1uKXwuzvY9k21lzeMoEcvdDc9GgTrZFqWq3gwWKI-yK3g0lh0oUOQ0vA==  pkg=pmapi
   6 Requesting GET /events/S19Mv-10ul4GzYUrBoxBjBhLk4e3wpwPYEhvkHVowwkM5puOrAjigLG_rAYWDIJ-zRcWZuWkKnKPYZmPh4bRwQ==  pkg=pmapi
   7 Requesting GET /events/Wtqohxyn7Da7HeR2AQjB9hCsbqvJMgWwjrYDrBJ1b57fyEcvFfdawarBm8-33exrhPPlV-W7Z4KXeOFc-37yiQ==  pkg=pmapi
  10 Requesting GET /events/HkvJk4Lbg6FTsSjqIKR-Rk6fDwY3k4kOR0UiMN5Y8hBEK6gMwFhUsyTv8AHtVj45MWsp9PaCQmoqAGI2buFq5g==  pkg=pmapi
  11 Requesting GET /events/LzYKC64TEkyESV4j58EMVx8vRMOKUOF3bv78oDvDd36ziwYmUQk4tgODZt_lhbDQHIDNMud1mQMRacFw1lJRcw==  pkg=pmapi
  18 Requesting GET /events/d7vA5EIcWRCoQ2lc7zzlu2gy0bJuyNi-zVllIT66GqEbP7yI5KW4hUy4MNYkgNq4IWSw5XEqyKljMSytes6J1Q==  pkg=pmapi
  40 Requesting GET /events/6Vj3-tIgRt32Vx8vOhjy1acS_XivgL5yp_InXpmLwtj2CGyoumdiK56TjJ3otgVZMZsGZlEBHYNI9M6KFZprHw==  pkg=pmapi

[andrzejsza]

reopening this as we're investigating. we were assuming the excessive traffic comes from client-side, but are now confirming this theory. will keep you posted.

[tomtastic]

Thanks @andrzejsza, let me know if I can gather any logs, etc to help.

[ianhundere]

+1

❯ du -sh .config/protonmail/bridge/cache/c11/ | sort
9.1G	.config/protonmail/bridge/cache/c11/

[andrzejsza]

we've completely change the behavior with Bridge v3. closing the issue, it anyone still experiencing unreasonably high network traffic - please get in touch with support for further debugging.