Transactions Not Syncing in Polled Mode Without Manual Refresh

[adaofeliz]

When operating in polled mode, I've encountered a consistent issue where transactions are not being automatically added to Firefly. This occurs despite waiting for periods extending beyond 24 hours. The transactions only appear after manually triggering a refresh using a cURL request to /transactions/refresh.

As a workaround, I am considering implementing an automated task to trigger /transactions/refresh every hour. However, this is not an ideal long-term solution, and I am seeking insights into the root cause and potential fixes.

• Has anyone else experienced similar issues with transactions not syncing in polled mode unless manually refreshed?
• Are there known issues or limitations with the /transactions/sync endpoint that could be causing this behaviour?
• Any recommendations on how to address this problem more effectively, or is there an existing patch or update that resolves this issue?

[dvankley]

I have not personally seen this issue.

Do you only see it with a single institution or all your institutions? My first guess is it's an issue on Plaid's end, failing to trigger updates on the sync endpoint regularly as they're supposed to. I would suggest contacting their support, especially if you're a paid user.

I'm not categorically opposed to building this feature in, although the connector doesn't currently have any means of accepting user input, and the pricing for the coming paid cutover looks like it would make this feature expensive.

[aacater]

I have been experiencing something similar, specifically with Discover (that is the only institution where I make transactions frequently enough to notice a problem). Transactions are only found inconsistently. I hadn't tracked down the exact cause yet.

I will try /transactions/refresh once I get access to it.

I did find this in the /transactions/sync docs (link).

Plaid typically checks for new data multiple times a day, but these checks may occur less frequently, such as once a day, depending on the institution. An Item's status.transactions.last_successful_update field will show the timestamp of the most recent successful update. To force Plaid to check for new transactions, use the /transactions/refresh endpoint.

Although when I make a request to /item/get/, it is shown to be sufficiently up to date. Forgot about timezones.

[adaofeliz]

I have been experiencing something similar, specifically with Discover (that is the only institution where I make transactions frequently enough to notice a problem). Transactions are only found inconsistently. I hadn't tracked down the exact cause yet.

I will try /transactions/refresh once I get access to it.

I did find this in the /transactions/sync docs (link).

Plaid typically checks for new data multiple times a day, but these checks may occur less frequently, such as once a day, depending on the institution. An Item's status.transactions.last_successful_update field will show the timestamp of the most recent successful update. To force Plaid to check for new transactions, use the /transactions/refresh endpoint.

Although when I make a request to /item/get/, it is shown to be sufficiently up to date.

Thank you for sharing your experiences and for referencing the relevant section of the Plaid documentation.

Indeed, the portion of the documentation you mentioned guided me to experiment with the /transactions/refresh endpoint. Upon triggering a manual refresh, I noticed that transactions synced as expected. This observation leads me to conclude that the issue might not stem from the firefly-plaid-connector-2 project or even Plaid's infrastructure directly. Rather, it appears the root cause could be related to how frequently my banking institution updates transaction data on Plaid's platform. It seems that proactive updates are not occurring, and Plaid's integration with my bank only effectively refreshes transaction data when explicitly prompted by the /transactions/refresh.

The /transactions/sync endpoint is well-designed for its intended purpose but may not serve effectively in my specific scenario due to the bank's update frequency. As a temporarry workaround to ensure consistent transaction updates, I plan to implement an hourly refresh using a cURL command in crontab. At least while it remains a free solution.

I'm considering developing a smart batch process, potentially leveraging a local SQLite database, to identify and sync missing transactions in Firefly while avoiding duplicates. If you have any suggestions or if there's a potential for collaboration on developing a more refined solution, I'm open to ideas.

[dvankley]

Indeed, the portion of the documentation you mentioned guided me to experiment with the /transactions/refresh endpoint. Upon triggering a manual refresh, I noticed that transactions synced as expected. This observation leads me to conclude that the issue might not stem from the firefly-plaid-connector-2 project or even Plaid's infrastructure directly. Rather, it appears the root cause could be related to how frequently my banking institution updates transaction data on Plaid's platform. It seems that proactive updates are not occurring, and Plaid's integration with my bank only effectively refreshes transaction data when explicitly prompted by the /transactions/refresh.

Yeah, I suppose it depends on Plaid's architecture for updating transactions for your particular institution. I assume they have a mix of pull and push models depending on the API for each institution. If a manual refresh works and the automatic, periodic refresh doesn't, I would infer that they use different methods (i.e. your bank pushes to Plaid automatically, manual refresh does a full pull on another endpoint).
Regardless, my reading of the Plaid docs is that Plaid should refresh data at minimum once a day; if so, there's definitely something wrong in your case somewhere between Plaid and your bank. Since Plaid's whole value proposition is handling the interface with banks, I would suggest taking this up with Plaid support so they can resolve it rather than trying to work around it.

The /transactions/sync endpoint is well-designed for its intended purpose but may not serve effectively in my specific scenario due to the bank's update frequency. As a temporarry workaround to ensure consistent transaction updates, I plan to implement an hourly refresh using a cURL command in crontab. At least while it remains a free solution.

Sure, that seems straightforward enough to do what you want. I'm also not opposed to adding something like that to the connector, but as previously mentioned I don't think it's going to have much long term value.

I'm considering developing a smart batch process, potentially leveraging a local SQLite database, to identify and sync missing transactions in Firefly while avoiding duplicates. If you have any suggestions or if there's a potential for collaboration on developing a more refined solution, I'm open to ideas.

I would need some more details on what you're trying to accomplish and how you're planning on doing it before I can really offer anything, but happy to talk about ideas.

[aacater]

my reading of the Plaid docs is that Plaid should refresh data at minimum once a day

I looked up the institutions I was having problems with on https://dashboard.plaid.com/activity/status and they either have limited availability based on the time of day or have a overall poor success rates.

So I would say this is definitely not a problem with the connector.

But as a work around, I wouldn't mind an option to automatically refresh every x amount of time. Although that might be more trouble than its worth. Its another thing to track. And you would have to account for institutions limited availability periods, there is no point in refreshing when the institution is limited. Although the billing for /transactions/refresh is per each successful call so there is that.

[aacater]

I will try /transactions/refresh once I get access to it.

I can confirm that /transactions/refresh enables the connector to properly pull transactions.

Although when I make a request to /item/get/, it is shown to be sufficiently up to date.

I didn't realize the endpoint reports time in UTC so I thought the institutions were up to date when in fact they weren't.

[brccabral]

I have noticed the same problem in my account (I currently only have one).
As this could be related to each account, we could have refresh settings in the application.yml at the account level.

fireflyPlaidConnector2:
  accounts:
    - fireflyAccountId: X
      plaidItemAccessToken: access-environment-xyz
      plaidAccountId: abc
      forceRefresh: true|false
      refreshInterval: 60 # minutes