Skip to content

[10.x] Fix Postgres cache store failed to put exist cache in transaction#48968

Merged
taylorotwell merged 10 commits into
laravel:10.xfrom
xdevor:fix-pgsql-cache-store-put-bug
Nov 13, 2023
Merged

[10.x] Fix Postgres cache store failed to put exist cache in transaction#48968
taylorotwell merged 10 commits into
laravel:10.xfrom
xdevor:fix-pgsql-cache-store-put-bug

Conversation

@xdevor

@xdevor xdevor commented Nov 10, 2023

Copy link
Copy Markdown
Contributor

Problem

When using Postgres as the Cache database store,
an error occurs when performing put operation on an already existing cache key during a transaction.
The error is as follows: SQLSTATE[25P02]: In failed sql transaction: 7 ERROR: current transaction is aborted, commands ignored until end of transaction block...

Steps To Reproduce

  • Step 1: after setup postgres and .env file put the following code to laravel routes/console.php
use Illuminate\Support\Facades\Cache;
use Illuminate\Support\Facades\DB;

Artisan::command('test', function () {

    $store = Cache::store('database');

    DB::beginTransaction();

    $store->put('test-key', 'test-value');

    DB::commit();

});
  • Step 2: run php artisan test twice times and the below error will show
Screen Shot 2023-11-10 at 10 39 39 PM

Versions

  • Postgres: v12 and v16
  • PHP: 8.2
  • Laravel: 10

@xdevor xdevor changed the title [10.x] Fix Postgres cache store failed to put in db transaction [10.x] Fix Postgres cache store failed to put exist cache in transaction Nov 10, 2023
Comment thread tests/Cache/CacheDatabaseStoreTest.php Outdated
@xdevor
xdevor force-pushed the fix-pgsql-cache-store-put-bug branch from fa22473 to b6b0c23 Compare November 11, 2023 02:29
@tpetry

tpetry commented Nov 11, 2023

Copy link
Copy Markdown
Contributor

Wouldn't an upsert for all database drivers make the most sense? For existing keys they will have to execute two queries instead of just one with upserts.

@xdevor

xdevor commented Nov 11, 2023

Copy link
Copy Markdown
Contributor Author

@tpetry 100% agree with your point. However, the reason I didn't implement it this way in this bug fix PR is because I want to minimize possible side effects. Also, I'm not entirely sure if there's a reason that led to the previous implementation not using upsert

@tpetry

tpetry commented Nov 11, 2023

Copy link
Copy Markdown
Contributor

@plehatron Was there a specific reason for not using upsert?

@xdevor

xdevor commented Nov 12, 2023

Copy link
Copy Markdown
Contributor Author

It seems there's no issue with using upsert for databases supported by Laravel. I've adjusted the statement.

@tpetry

tpetry commented Nov 13, 2023

Copy link
Copy Markdown
Contributor

The add method has the same problem with PG when the insert fails because of a duplicate key. But honestly, I don't understand the code in the exception case. The logic seems to be different than described in the docblock.

@plehatron

Copy link
Copy Markdown
Contributor

@plehatron Was there a specific reason for not using upsert?

Hey @tpetry, the original logic for DatabaseStore::put predates my work from #26726 where I made sure the cache component is fully compliant with the PSR-16 SimpleCache interface. Maybe @taylorotwell has more insight into why upsert was not used then 🤷‍♂️.

@taylorotwell
taylorotwell merged commit c6aeffd into laravel:10.x Nov 13, 2023
@xdevor

xdevor commented Nov 14, 2023

Copy link
Copy Markdown
Contributor Author

The add method has the same problem with PG when the insert fails because of a duplicate key. But honestly, I don't understand the code in the exception case. The logic seems to be different than described in the docblock.

@tpetry seems like the add() method is designed to ensure that the cache value is not overwritten when the cache exists. However, it is challenging to prevent race conditions without lock (and lock has risk), that's why the current implementation like this.

Perhaps the only solution for now is to delete expired caches before each time add() call or periodically delete expired cache in laravel project.

@tpetry

tpetry commented Nov 14, 2023

Copy link
Copy Markdown
Contributor

@tpetry seems like the add() method is designed to ensure that the cache value is not overwritten when the cache exists. However, it is challenging to prevent race conditions without lock (and lock has risk), that's why the current implementation like this.

But the logic in the catch-part is overwriting any value when the insert fails with an exception due to an existing value?

@xdevor

xdevor commented Nov 14, 2023

Copy link
Copy Markdown
Contributor Author

But the logic in the catch-part is overwriting any value when the insert fails with an exception due to an existing value?

Because expired cache (where('expiration', '<=', $this->getTime()) ) is to be considered as not existing cache, and db will not remove the expired record automatically.

} catch (QueryException) {
        return $this->table()
            ->where('key', $key)
             ->where('expiration', '<=', $this->getTime()) // <--- only update expired cache, and expired cache is to be considered as not existing
            ->update([
                'value' => $value,
                'expiration' => $expiration,
            ]) >= 1;
}

@tpetry

tpetry commented Nov 14, 2023

Copy link
Copy Markdown
Contributor

Oh you're right. didn't look could enough. Still the current transaction is aborted would also happen if the insert fails, an insertOrIgnore needs to be used here.

@xdevor

xdevor commented Nov 14, 2023

Copy link
Copy Markdown
Contributor Author

Oh you're right. didn't look could enough. Still the current transaction is aborted would also happen if the insert fails, an insertOrIgnore needs to be used here.

@tpetry seems like possible solution ~ maybe implement like this to make sure it will not be silent while race condition occur, and call get() before insert because get() will delete expired cache automatically

// if connection is postgres and in transaction:
throw_if(
    is_null($this->table()->get($key))
    && $this->table()->insertOrIgnore(compact('key', 'value', 'expiration')) === 0,
    new \RuntimeException("Cache key '$key' has been set.")
)
return true;

UPDATE

or like this, because the current implementation of add() actually have overwrite problem while race condition occur in its error handle statement, also it keep silent while duplicate key exist, so we don't need to throw exception and should only do insert operation without any update in error handle

// for all database
return is_null($this->table()->get($key))
    && $this->table()->insertOrIgnore(compact('key', 'value', 'expiration')) > 0;

this approach fix two problem

  • one is postgres error in transaction
  • another one is overwritten while race condition occur in error handle statement

@tpetry you can open a PR for this 👍 , if you have no time or instrest I can do this

@xdevor
xdevor deleted the fix-pgsql-cache-store-put-bug branch November 16, 2023 23:21
renovate Bot referenced this pull request in RadioRoster/backend Nov 24, 2023
[![Mend Renovate logo
banner](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Change | Age | Adoption | Passing | Confidence |
|---|---|---|---|---|---|
| [laravel/framework](https://laravel.com)
([source](https://togithub.com/laravel/framework)) | `10.30.1` ->
`10.33.0` |
[![age](https://developer.mend.io/api/mc/badges/age/packagist/laravel%2fframework/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![adoption](https://developer.mend.io/api/mc/badges/adoption/packagist/laravel%2fframework/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![passing](https://developer.mend.io/api/mc/badges/compatibility/packagist/laravel%2fframework/10.30.1/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|
[![confidence](https://developer.mend.io/api/mc/badges/confidence/packagist/laravel%2fframework/10.30.1/10.33.0?slim=true)](https://docs.renovatebot.com/merge-confidence/)
|

---

### Release Notes

<details>
<summary>laravel/framework (laravel/framework)</summary>

###
[`v10.33.0`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10330---2023-11-21)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.32.1...v10.33.0)

- \[10.x] Fix wrong parameter passing and add these rules to dependent
rules by [@&#8203;kayw-geek](https://togithub.com/kayw-geek) in
[https://github.com/laravel/framework/pull/49008](https://togithub.com/laravel/framework/pull/49008)
- \[10.x] Make Validator::getValue() public by
[@&#8203;shinsenter](https://togithub.com/shinsenter) in
[https://github.com/laravel/framework/pull/49007](https://togithub.com/laravel/framework/pull/49007)
- \[10.x] Custom messages for `Password` validation rule by
[@&#8203;rcknr](https://togithub.com/rcknr) in
[https://github.com/laravel/framework/pull/48928](https://togithub.com/laravel/framework/pull/48928)
- \[10.x] Round milliseconds in database seeder console output runtime
by [@&#8203;SjorsO](https://togithub.com/SjorsO) in
[https://github.com/laravel/framework/pull/49014](https://togithub.com/laravel/framework/pull/49014)
- \[10.x] Add a `Number` utility class by
[@&#8203;caendesilva](https://togithub.com/caendesilva) in
[https://github.com/laravel/framework/pull/48845](https://togithub.com/laravel/framework/pull/48845)
- \[10.x] Fix the replace() method in DefaultService class by
[@&#8203;jonagoldman](https://togithub.com/jonagoldman) in
[https://github.com/laravel/framework/pull/49022](https://togithub.com/laravel/framework/pull/49022)
- \[10.x] Pass the property $validator as a parameter to the $callback
Closure by [@&#8203;shinsenter](https://togithub.com/shinsenter) in
[https://github.com/laravel/framework/pull/49015](https://togithub.com/laravel/framework/pull/49015)
- \[10.x] Fix Cache DatabaseStore::add() error occur on Postgres within
transaction by [@&#8203;xdevor](https://togithub.com/xdevor) in
[https://github.com/laravel/framework/pull/49025](https://togithub.com/laravel/framework/pull/49025)
- \[10.x] Support asserting against chained batches by
[@&#8203;taylorotwell](https://togithub.com/taylorotwell) in
[https://github.com/laravel/framework/pull/49003](https://togithub.com/laravel/framework/pull/49003)
- \[10.x] Prevent DB `Cache::get()` occur race condition by
[@&#8203;xdevor](https://togithub.com/xdevor) in
[https://github.com/laravel/framework/pull/49031](https://togithub.com/laravel/framework/pull/49031)
- \[10.x] Fix notifications being counted as sent without a "shouldSend"
method by [@&#8203;joelwmale](https://togithub.com/joelwmale) in
[https://github.com/laravel/framework/pull/49030](https://togithub.com/laravel/framework/pull/49030)
- \[10.x] Fix tests failure on Windows by
[@&#8203;hafezdivandari](https://togithub.com/hafezdivandari) in
[https://github.com/laravel/framework/pull/49037](https://togithub.com/laravel/framework/pull/49037)
- \[10.x] Add unless conditional on validation rules by
[@&#8203;michaelnabil230](https://togithub.com/michaelnabil230) in
[https://github.com/laravel/framework/pull/49048](https://togithub.com/laravel/framework/pull/49048)
- \[10.x] Handle string based payloads that are not JSON or form data
when creating PSR request instances by
[@&#8203;timacdonald](https://togithub.com/timacdonald) in
[https://github.com/laravel/framework/pull/49047](https://togithub.com/laravel/framework/pull/49047)
- \[10.x] Fix directory separator CMD display on windows by
[@&#8203;imanghafoori1](https://togithub.com/imanghafoori1) in
[https://github.com/laravel/framework/pull/49045](https://togithub.com/laravel/framework/pull/49045)
- \[10.x] Fix mapSpread doc by
[@&#8203;timacdonald](https://togithub.com/timacdonald) in
[https://github.com/laravel/framework/pull/48941](https://togithub.com/laravel/framework/pull/48941)
- \[10.x] Tiny `Support\Collection` test fix - Unused data provider
parameter by [@&#8203;stevebauman](https://togithub.com/stevebauman) in
[https://github.com/laravel/framework/pull/49053](https://togithub.com/laravel/framework/pull/49053)
- \[10.x] Feat: Add color_hex validation rule by
[@&#8203;nikopeikrishvili](https://togithub.com/nikopeikrishvili) in
[https://github.com/laravel/framework/pull/49056](https://togithub.com/laravel/framework/pull/49056)
- \[10.x] Handle missing translation strings using callback by
[@&#8203;DeanWunder](https://togithub.com/DeanWunder) in
[https://github.com/laravel/framework/pull/49040](https://togithub.com/laravel/framework/pull/49040)
- \[10.x] Add Str::transliterate to Stringable by
[@&#8203;dwightwatson](https://togithub.com/dwightwatson) in
[https://github.com/laravel/framework/pull/49065](https://togithub.com/laravel/framework/pull/49065)
- Add Alpha Channel support to Hex validation rule by
[@&#8203;ahinkle](https://togithub.com/ahinkle) in
[https://github.com/laravel/framework/pull/49069](https://togithub.com/laravel/framework/pull/49069)

###
[`v10.32.1`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10321---2023-11-14)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.32.0...v10.32.1)

- \[10.x] Add `[@pushElseIf](https://togithub.com/pushElseIf)` and
`[@pushElse](https://togithub.com/pushElse)` by
[@&#8203;jasonmccreary](https://togithub.com/jasonmccreary) in
[https://github.com/laravel/framework/pull/48990](https://togithub.com/laravel/framework/pull/48990)

###
[`v10.32.0`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10320---2023-11-14)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.31.0...v10.32.0)

- Update PendingRequest.php by
[@&#8203;mattkingshott](https://togithub.com/mattkingshott) in
[https://github.com/laravel/framework/pull/48939](https://togithub.com/laravel/framework/pull/48939)
- \[10.x] Change array_key_exists with null coalescing assignment
operator in FilesystemAdapter by
[@&#8203;miladev95](https://togithub.com/miladev95) in
[https://github.com/laravel/framework/pull/48943](https://togithub.com/laravel/framework/pull/48943)
- \[10.x] Use container to resolve email validator class by
[@&#8203;orkhanahmadov](https://togithub.com/orkhanahmadov) in
[https://github.com/laravel/framework/pull/48942](https://togithub.com/laravel/framework/pull/48942)
- \[10.x] Added `getGlobalMiddleware` method to HTTP Client Factory by
[@&#8203;pascalbaljet](https://togithub.com/pascalbaljet) in
[https://github.com/laravel/framework/pull/48950](https://togithub.com/laravel/framework/pull/48950)
- \[10.x] Detect MySQL read-only mode error as a lost connection by
[@&#8203;cosmastech](https://togithub.com/cosmastech) in
[https://github.com/laravel/framework/pull/48937](https://togithub.com/laravel/framework/pull/48937)
- \[10.x] Adds more implicit validation rules for `present` based on
other fields by
[@&#8203;diamondobama](https://togithub.com/diamondobama) in
[https://github.com/laravel/framework/pull/48908](https://togithub.com/laravel/framework/pull/48908)
- \[10.x] Refactor set_error_handler callback to use arrow function in
`InteractsWithDeprecationHandling` by
[@&#8203;miladev95](https://togithub.com/miladev95) in
[https://github.com/laravel/framework/pull/48954](https://togithub.com/laravel/framework/pull/48954)
- \[10.x] Test Improvements by
[@&#8203;crynobone](https://togithub.com/crynobone) in
[https://github.com/laravel/framework/pull/48962](https://togithub.com/laravel/framework/pull/48962)
- Fix issue that prevents BladeCompiler to raise an exception when
temporal compiled blade template is not found. by
[@&#8203;juanparati](https://togithub.com/juanparati) in
[https://github.com/laravel/framework/pull/48957](https://togithub.com/laravel/framework/pull/48957)
- \[10.x] Fix how nested transaction callbacks are handled by
[@&#8203;mateusjatenee](https://togithub.com/mateusjatenee) in
[https://github.com/laravel/framework/pull/48859](https://togithub.com/laravel/framework/pull/48859)
- \[10.x] Fixes Batch Callbacks not triggering if job timeout while in
transaction by [@&#8203;crynobone](https://togithub.com/crynobone) in
[https://github.com/laravel/framework/pull/48961](https://togithub.com/laravel/framework/pull/48961)
- \[10.x] expressions in migration computations fail by
[@&#8203;tpetry](https://togithub.com/tpetry) in
[https://github.com/laravel/framework/pull/48976](https://togithub.com/laravel/framework/pull/48976)
- \[10.x] Fixes Exception: Cannot traverse an already closed generator
when running Arr::first with an empty generator and no callback by
[@&#8203;moshe-autoleadstar](https://togithub.com/moshe-autoleadstar) in
[https://github.com/laravel/framework/pull/48979](https://togithub.com/laravel/framework/pull/48979)
- fixes issue with stderr when there was "]" character. by
[@&#8203;nikopeikrishvili](https://togithub.com/nikopeikrishvili) in
[https://github.com/laravel/framework/pull/48975](https://togithub.com/laravel/framework/pull/48975)
- \[10.x] Fix Postgres cache store failed to put exist cache in
transaction by [@&#8203;xdevor](https://togithub.com/xdevor) in
[https://github.com/laravel/framework/pull/48968](https://togithub.com/laravel/framework/pull/48968)

###
[`v10.31.0`](https://togithub.com/laravel/framework/blob/HEAD/CHANGELOG.md#v10310---2023-11-07)

[Compare
Source](https://togithub.com/laravel/framework/compare/v10.30.1...v10.31.0)

- \[10.x] Allow `Sleep::until()` to be passed a timestamp as a string by
[@&#8203;jameshulse](https://togithub.com/jameshulse) in
[https://github.com/laravel/framework/pull/48883](https://togithub.com/laravel/framework/pull/48883)
- \[10.x] Fix whereHasMorph() with nullable morphs by
[@&#8203;MarkKremer](https://togithub.com/MarkKremer) in
[https://github.com/laravel/framework/pull/48903](https://togithub.com/laravel/framework/pull/48903)
- \[10.x] Handle `class_parents` returning false in
`class_uses_recursive` by
[@&#8203;RoflCopter24](https://togithub.com/RoflCopter24) in
[https://github.com/laravel/framework/pull/48902](https://togithub.com/laravel/framework/pull/48902)
- \[10.x] Enable default retrieval of all fragments in `fragments()` and
`fragmentsIf()` methods by [@&#8203;tabuna](https://togithub.com/tabuna)
in
[https://github.com/laravel/framework/pull/48894](https://togithub.com/laravel/framework/pull/48894)
- \[10.x] Allow placing a batch on a chain by
[@&#8203;khepin](https://togithub.com/khepin) in
[https://github.com/laravel/framework/pull/48633](https://togithub.com/laravel/framework/pull/48633)
- \[10.x] Dispatch 'connection failed' event in async http client
request by [@&#8203;gdebrauwer](https://togithub.com/gdebrauwer) in
[https://github.com/laravel/framework/pull/48900](https://togithub.com/laravel/framework/pull/48900)
- authenticate method refactored to use null coalescing operator by
[@&#8203;miladev95](https://togithub.com/miladev95) in
[https://github.com/laravel/framework/pull/48917](https://togithub.com/laravel/framework/pull/48917)
- \[10.x] Add support for Sec-Purpose header by
[@&#8203;nanos](https://togithub.com/nanos) in
[https://github.com/laravel/framework/pull/48925](https://togithub.com/laravel/framework/pull/48925)
- \[10.x] Allow setting retain_visibility config option on Flysystem
filesystems by [@&#8203;jnoordsij](https://togithub.com/jnoordsij) in
[https://github.com/laravel/framework/pull/48935](https://togithub.com/laravel/framework/pull/48935)
- \[10.x] Escape forward slashes when exploding wildcard rules by
[@&#8203;matt-farrugia](https://togithub.com/matt-farrugia) in
[https://github.com/laravel/framework/pull/48936](https://togithub.com/laravel/framework/pull/48936)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined),
Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR is behind base branch, or you tick the
rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update
again.

---

- [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check
this box

---

This PR has been generated by [Mend
Renovate](https://www.mend.io/free-developer-tools/renovate/). View
repository job log
[here](https://developer.mend.io/github/Lapotor/RadioRoster-api).

<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNy41OS44IiwidXBkYXRlZEluVmVyIjoiMzcuNTkuOCIsInRhcmdldEJyYW5jaCI6Im1haW4ifQ==-->

Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants