Fix/Do not return expired objects #1634
Conversation
carpawell
added
bug
Codecov Report
@@ Coverage Diff @@
## master #1634 +/- ##
==========================================
+ Coverage 33.17% 33.19% +0.01%
==========================================
Files 332 332
Lines 22709 22744 +35
==========================================
+ Hits 7534 7549 +15
- Misses 14557 14576 +19
- Partials 618 619 +1
📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more |
| expirationBucket := tx.Bucket(attributeBucketName(addr.Container(), objectV2.SysAttributeExpEpoch)) | ||
| if expirationBucket != nil { | ||
| // bucket that contains objects that expire in the current epoch | ||
| currEpochBkt := expirationBucket.Bucket([]byte(strconv.FormatUint(currEpoch, 10))) |
There was a problem hiding this comment.
What about the objects that expired in the previous epoch?
There was a problem hiding this comment.
i've added comment to that func:
// we check only if the object is expired in the current
// epoch since it is considered the only corner case: the
// GC is expected to collect all the objects that have
// expired previously for less than the one epoch duration
i expect that GC is able to collect all the expired objects in one epoch (one hour currently), so the only problem objects are the object that expire in the current epoch but have not been collected by the GC yet. theoretically, we could check more than one epoch (we do not have index that could say what objects are expired, only the objects that has exact expiration epoch), so we should know how deep to check OR we should add new expiration index
i thought about force new epoch event that could lead to some problems. we may add some threshold for that cases (e.g. check if an object was not expired in the last 5 epoch)
what do you think?
| @@ -89,7 +90,7 @@ func (db *DB) delete(tx *bbolt.Tx, addr oid.Address, refCounter referenceCounter | |||
| } | |||
|
|
|||
| // unmarshal object, work only with physically stored (raw == true) objects | |||
| obj, err := db.get(tx, addr, false, true) | |||
| obj, err := db.get(tx, addr, false, true, currEpoch) | |||
There was a problem hiding this comment.
How do we ever delete an expired object from the metabase?
There was a problem hiding this comment.
third arg (false here) controls checking object status. if an object is presented, it will be deleted, despite its status
carpawell
requested review from
cthulhu-rider,
fyrchik,
realloc and
KirillovDenis
|
Resolve conflicts please |
It allows performing expiration checks on the stored objects. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
If an object has not been marked for removal by the GC in the current epoch yet but has already expired, respond with `ErrObjectNotFound` api status. Also, optimize shard iteration: a node must stop any iteration if the object is found but gonna be removed soon. All the checks are performed by the Metabase. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
It allows performing expiration checks on the stored objects. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Do not treat objects with expiration as expired by default. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
It allows performing expiration checks on the stored objects. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
If an object has not been marked for removal by the GC in the current epoch yet but has already expired, respond with `ErrObjectNotFound` api status. Also, optimize shard iteration: a node must stop any iteration if the object is found but gonna be removed soon. All the checks are performed by the Metabase. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Do not treat objects with expiration as expired by default. Signed-off-by: Pavel Karpy <carpawell@nspcc.ru>
Closes #1617.