Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[10.x] Fix Postgres cache store failed to put exist cache in transaction #48968

Merged
merged 10 commits into from Nov 13, 2023

Conversation

xdevor
Copy link
Contributor

@xdevor xdevor commented Nov 10, 2023

Problem

When using Postgres as the Cache database store,
an error occurs when performing put operation on an already existing cache key during a transaction.
The error is as follows: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR: current transaction is aborted, commands ignored until end of transaction block...

Steps To Reproduce

  • Step 1: after setup postgres and .env file put the following code to laravel routes/console.php
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\DB;

Artisan::command('test', function () {

    $store = Cache::store('database');

    DB::beginTransaction();

    $store->put('test-key', 'test-value');

    DB::commit();

});
  • Step 2: run php artisan test twice times and the below error will show
Screen Shot 2023-11-10 at 10 39 39 PM

Versions

  • Postgres: v12 and v16
  • PHP: 8.2
  • Laravel: 10

@xdevor xdevor changed the title [10.x] Fix Postgres cache store failed to put in db transaction [10.x] Fix Postgres cache store failed to put exist cache in transaction Nov 10, 2023
@xdevor xdevor force-pushed the fix-pgsql-cache-store-put-bug branch from fa22473 to b6b0c23 Compare November 11, 2023 02:29
@tpetry
Copy link
Contributor

tpetry commented Nov 11, 2023

Wouldn't an upsert for all database drivers make the most sense? For existing keys they will have to execute two queries instead of just one with upserts.

@xdevor
Copy link
Contributor Author

xdevor commented Nov 11, 2023

@tpetry 100% agree with your point. However, the reason I didn't implement it this way in this bug fix PR is because I want to minimize possible side effects. Also, I'm not entirely sure if there's a reason that led to the previous implementation not using upsert

@tpetry
Copy link
Contributor

tpetry commented Nov 11, 2023

@plehatron Was there a specific reason for not using upsert?

@xdevor
Copy link
Contributor Author

xdevor commented Nov 12, 2023

It seems there's no issue with using upsert for databases supported by Laravel. I've adjusted the statement.

@tpetry
Copy link
Contributor

tpetry commented Nov 13, 2023

The add method has the same problem with PG when the insert fails because of a duplicate key. But honestly, I don't understand the code in the exception case. The logic seems to be different than described in the docblock.

@plehatron
Copy link
Contributor

@plehatron Was there a specific reason for not using upsert?

Hey @tpetry, the original logic for DatabaseStore::put predates my work from #26726 where I made sure the cache component is fully compliant with the PSR-16 SimpleCache interface. Maybe @taylorotwell has more insight into why upsert was not used then 🤷‍♂️.

@taylorotwell taylorotwell merged commit c6aeffd into laravel:10.x Nov 13, 2023
20 checks passed
@xdevor
Copy link
Contributor Author

xdevor commented Nov 14, 2023

The add method has the same problem with PG when the insert fails because of a duplicate key. But honestly, I don't understand the code in the exception case. The logic seems to be different than described in the docblock.

@tpetry seems like the add() method is designed to ensure that the cache value is not overwritten when the cache exists. However, it is challenging to prevent race conditions without lock (and lock has risk), that's why the current implementation like this.

Perhaps the only solution for now is to delete expired caches before each time add() call or periodically delete expired cache in laravel project.

@tpetry
Copy link
Contributor

tpetry commented Nov 14, 2023

@tpetry seems like the add() method is designed to ensure that the cache value is not overwritten when the cache exists. However, it is challenging to prevent race conditions without lock (and lock has risk), that's why the current implementation like this.

But the logic in the catch-part is overwriting any value when the insert fails with an exception due to an existing value?

@xdevor
Copy link
Contributor Author

xdevor commented Nov 14, 2023

But the logic in the catch-part is overwriting any value when the insert fails with an exception due to an existing value?

Because expired cache (where('expiration', '<=', $this->getTime()) ) is to be considered as not existing cache, and db will not remove the expired record automatically.

} catch (QueryException) {
        return $this->table()
            ->where('key', $key)
             ->where('expiration', '<=', $this->getTime()) // <--- only update expired cache, and expired cache is to be considered as not existing
            ->update([
                'value' => $value,
                'expiration' => $expiration,
            ]) >= 1;
}

@tpetry
Copy link
Contributor

tpetry commented Nov 14, 2023

Oh you're right. didn't look could enough. Still the current transaction is aborted would also happen if the insert fails, an insertOrIgnore needs to be used here.

@xdevor
Copy link
Contributor Author

xdevor commented Nov 14, 2023

Oh you're right. didn't look could enough. Still the current transaction is aborted would also happen if the insert fails, an insertOrIgnore needs to be used here.

@tpetry seems like possible solution ~ maybe implement like this to make sure it will not be silent while race condition occur, and call get() before insert because get() will delete expired cache automatically

// if connection is postgres and in transaction:
throw_if(
    is_null($this->table()->get($key))
    && $this->table()->insertOrIgnore(compact('key', 'value', 'expiration')) === 0,
    new \RuntimeException("Cache key '$key' has been set.")
)
return true;

UPDATE

or like this, because the current implementation of add() actually have overwrite problem while race condition occur in its error handle statement, also it keep silent while duplicate key exist, so we don't need to throw exception and should only do insert operation without any update in error handle

// for all database
return is_null($this->table()->get($key))
    && $this->table()->insertOrIgnore(compact('key', 'value', 'expiration')) > 0;

this approach fix two problem

  • one is postgres error in transaction
  • another one is overwritten while race condition occur in error handle statement

@tpetry you can open a PR for this 👍 , if you have no time or instrest I can do this

@xdevor xdevor deleted the fix-pgsql-cache-store-put-bug branch November 16, 2023 23:21
renovate bot added a commit to RadioRoster/backend that referenced this pull request Nov 24, 2023
[![Mend Renovate logo
banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [laravel/framework](https://laravel.com)
([source](https://togithub.com/laravel/framework)) | `10.30.1` ->
`10.33.0` |
[![age](https://developer.mend.io/api/mc/badges/age/packagist/laravel%2fframework/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/packagist/laravel%2fframework/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/packagist/laravel%2fframework/10.30.1/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/packagist/laravel%2fframework/10.30.1/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>laravel/framework (laravel/framework)</summary>

###
[`v10.33.0`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10330---2023-11-21)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.32.1...v10.33.0)

- \[10.x] Fix wrong parameter passing and add these rules to dependent
rules by [@&#8203;kayw-geek](https://togithub.com/kayw-geek) in
[laravel/framework#49008
- \[10.x] Make Validator::getValue() public by
[@&#8203;shinsenter](https://togithub.com/shinsenter) in
[laravel/framework#49007
- \[10.x] Custom messages for `Password` validation rule by
[@&#8203;rcknr](https://togithub.com/rcknr) in
[laravel/framework#48928
- \[10.x] Round milliseconds in database seeder console output runtime
by [@&#8203;SjorsO](https://togithub.com/SjorsO) in
[laravel/framework#49014
- \[10.x] Add a `Number` utility class by
[@&#8203;caendesilva](https://togithub.com/caendesilva) in
[laravel/framework#48845
- \[10.x] Fix the replace() method in DefaultService class by
[@&#8203;jonagoldman](https://togithub.com/jonagoldman) in
[laravel/framework#49022
- \[10.x] Pass the property $validator as a parameter to the $callback
Closure by [@&#8203;shinsenter](https://togithub.com/shinsenter) in
[laravel/framework#49015
- \[10.x] Fix Cache DatabaseStore::add() error occur on Postgres within
transaction by [@&#8203;xdevor](https://togithub.com/xdevor) in
[laravel/framework#49025
- \[10.x] Support asserting against chained batches by
[@&#8203;taylorotwell](https://togithub.com/taylorotwell) in
[laravel/framework#49003
- \[10.x] Prevent DB `Cache::get()` occur race condition by
[@&#8203;xdevor](https://togithub.com/xdevor) in
[laravel/framework#49031
- \[10.x] Fix notifications being counted as sent without a "shouldSend"
method by [@&#8203;joelwmale](https://togithub.com/joelwmale) in
[laravel/framework#49030
- \[10.x] Fix tests failure on Windows by
[@&#8203;hafezdivandari](https://togithub.com/hafezdivandari) in
[laravel/framework#49037
- \[10.x] Add unless conditional on validation rules by
[@&#8203;michaelnabil230](https://togithub.com/michaelnabil230) in
[laravel/framework#49048
- \[10.x] Handle string based payloads that are not JSON or form data
when creating PSR request instances by
[@&#8203;timacdonald](https://togithub.com/timacdonald) in
[laravel/framework#49047
- \[10.x] Fix directory separator CMD display on windows by
[@&#8203;imanghafoori1](https://togithub.com/imanghafoori1) in
[laravel/framework#49045
- \[10.x] Fix mapSpread doc by
[@&#8203;timacdonald](https://togithub.com/timacdonald) in
[laravel/framework#48941
- \[10.x] Tiny `Support\Collection` test fix - Unused data provider
parameter by [@&#8203;stevebauman](https://togithub.com/stevebauman) in
[laravel/framework#49053
- \[10.x] Feat: Add color_hex validation rule by
[@&#8203;nikopeikrishvili](https://togithub.com/nikopeikrishvili) in
[laravel/framework#49056
- \[10.x] Handle missing translation strings using callback by
[@&#8203;DeanWunder](https://togithub.com/DeanWunder) in
[laravel/framework#49040
- \[10.x] Add Str::transliterate to Stringable by
[@&#8203;dwightwatson](https://togithub.com/dwightwatson) in
[laravel/framework#49065
- Add Alpha Channel support to Hex validation rule by
[@&#8203;ahinkle](https://togithub.com/ahinkle) in
[laravel/framework#49069

###
[`v10.32.1`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10321---2023-11-14)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.32.0...v10.32.1)

- \[10.x] Add `[@pushElseIf](https://togithub.com/pushElseIf)` and
`[@pushElse](https://togithub.com/pushElse)` by
[@&#8203;jasonmccreary](https://togithub.com/jasonmccreary) in
[laravel/framework#48990

###
[`v10.32.0`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10320---2023-11-14)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.31.0...v10.32.0)

- Update PendingRequest.php by
[@&#8203;mattkingshott](https://togithub.com/mattkingshott) in
[laravel/framework#48939
- \[10.x] Change array_key_exists with null coalescing assignment
operator in FilesystemAdapter by
[@&#8203;miladev95](https://togithub.com/miladev95) in
[laravel/framework#48943
- \[10.x] Use container to resolve email validator class by
[@&#8203;orkhanahmadov](https://togithub.com/orkhanahmadov) in
[laravel/framework#48942
- \[10.x] Added `getGlobalMiddleware` method to HTTP Client Factory by
[@&#8203;pascalbaljet](https://togithub.com/pascalbaljet) in
[laravel/framework#48950
- \[10.x] Detect MySQL read-only mode error as a lost connection by
[@&#8203;cosmastech](https://togithub.com/cosmastech) in
[laravel/framework#48937
- \[10.x] Adds more implicit validation rules for `present` based on
other fields by
[@&#8203;diamondobama](https://togithub.com/diamondobama) in
[laravel/framework#48908
- \[10.x] Refactor set_error_handler callback to use arrow function in
`InteractsWithDeprecationHandling` by
[@&#8203;miladev95](https://togithub.com/miladev95) in
[laravel/framework#48954
- \[10.x] Test Improvements by
[@&#8203;crynobone](https://togithub.com/crynobone) in
[laravel/framework#48962
- Fix issue that prevents BladeCompiler to raise an exception when
temporal compiled blade template is not found. by
[@&#8203;juanparati](https://togithub.com/juanparati) in
[laravel/framework#48957
- \[10.x] Fix how nested transaction callbacks are handled by
[@&#8203;mateusjatenee](https://togithub.com/mateusjatenee) in
[laravel/framework#48859
- \[10.x] Fixes Batch Callbacks not triggering if job timeout while in
transaction by [@&#8203;crynobone](https://togithub.com/crynobone) in
[laravel/framework#48961
- \[10.x] expressions in migration computations fail by
[@&#8203;tpetry](https://togithub.com/tpetry) in
[laravel/framework#48976
- \[10.x] Fixes Exception: Cannot traverse an already closed generator
when running Arr::first with an empty generator and no callback by
[@&#8203;moshe-autoleadstar](https://togithub.com/moshe-autoleadstar) in
[laravel/framework#48979
- fixes issue with stderr when there was "]" character. by
[@&#8203;nikopeikrishvili](https://togithub.com/nikopeikrishvili) in
[laravel/framework#48975
- \[10.x] Fix Postgres cache store failed to put exist cache in
transaction by [@&#8203;xdevor](https://togithub.com/xdevor) in
[laravel/framework#48968

###
[`v10.31.0`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10310---2023-11-07)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.30.1...v10.31.0)

- \[10.x] Allow `Sleep::until()` to be passed a timestamp as a string by
[@&#8203;jameshulse](https://togithub.com/jameshulse) in
[laravel/framework#48883
- \[10.x] Fix whereHasMorph() with nullable morphs by
[@&#8203;MarkKremer](https://togithub.com/MarkKremer) in
[laravel/framework#48903
- \[10.x] Handle `class_parents` returning false in
`class_uses_recursive` by
[@&#8203;RoflCopter24](https://togithub.com/RoflCopter24) in
[laravel/framework#48902
- \[10.x] Enable default retrieval of all fragments in `fragments()` and
`fragmentsIf()` methods by [@&#8203;tabuna](https://togithub.com/tabuna)
in
[laravel/framework#48894
- \[10.x] Allow placing a batch on a chain by
[@&#8203;khepin](https://togithub.com/khepin) in
[laravel/framework#48633
- \[10.x] Dispatch 'connection failed' event in async http client
request by [@&#8203;gdebrauwer](https://togithub.com/gdebrauwer) in
[laravel/framework#48900
- authenticate method refactored to use null coalescing operator by
[@&#8203;miladev95](https://togithub.com/miladev95) in
[laravel/framework#48917
- \[10.x] Add support for Sec-Purpose header by
[@&#8203;nanos](https://togithub.com/nanos) in
[laravel/framework#48925
- \[10.x] Allow setting retain_visibility config option on Flysystem
filesystems by [@&#8203;jnoordsij](https://togithub.com/jnoordsij) in
[laravel/framework#48935
- \[10.x] Escape forward slashes when exploding wildcard rules by
[@&#8203;matt-farrugia](https://togithub.com/matt-farrugia) in
[laravel/framework#48936

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/Lapotor/RadioRoster-api).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy41OS44IiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants