Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

read_batch raises a DataError with as_of a deleted version #1385

Closed
DrNickClarke opened this issue Feb 29, 2024 · 0 comments · Fixed by #1419
Closed

read_batch raises a DataError with as_of a deleted version #1385

DrNickClarke opened this issue Feb 29, 2024 · 0 comments · Fixed by #1419
Assignees
Labels
Bug fixing drive Q1 2024 bug Something isn't working

Comments

@DrNickClarke
Copy link
Collaborator

Describe the bug

When reading a version number that has been deleted a low level data error is returned. Also a warning is logged

[arcticdb] [warning] Failed to find segment for key 'i:asof_slow_read:499:0x1f40d440a9ebd654@1709222092432355772[1704067200000000000,1704240000000000001]' : No response body.

The error exception_string is
arcticdb::storage::KeyNotFoundException: Composite: i:asof_slow_read:499:0x1f40d440a9ebd654@1709222092432355772[1704067200000000000,1704240000000000001],

Steps/Code to Reproduce

Steps/Code to Reproduce

import arcticdb as adb
import pandas as pd
import numpy as np
from datetime import datetime, timedelta

arctic = adb.Arctic()
lib = arctic.get_library('adb_bugs', create_if_missing=True)

N = 3
df = pd.DataFrame(
index=pd.date_range("20240101", periods=N),
data={'col': np.arange(0., N)}
)

write 500 versions

sym1 = 'asof_slow_read'
for i in range(500):
lib.write(sym1, df)

remove early versions

lib.delete(sym1)

add one more version

lib.write(sym1, df)

this gives the error

v = lib.read_batch([adb.ReadRequest(sym1, as_of=499)])
v[0].exception_string

Expected Results

Error should be

  • Version not found
  • The same as for lib.read()
  • Not refer to low level storage issues

OS, Python Version and ArcticDB Version

Python 3.10
Linux Linux version 5.15.133.1-microsoft-standard-WSL2
arcticdb 4.3.1

Backend storage used

AWS S3

Additional Context

This is possibly related to this issue (failure to observe tombstone correctly). It may be easier to solve the two together

#1384

@DrNickClarke DrNickClarke added the bug Something isn't working label Feb 29, 2024
@alexowens90 alexowens90 modified the milestone: Bug fixing drive Q1 2024 Mar 14, 2024
muhammadhamzasajjad added a commit that referenced this issue Mar 15, 2024
…sion (#1419)

Fixes #1385 
Similar to a normal read, `batch_read` now sets `include_deleted` to
false when looking for an index key via `find_index_key_for_version_id`
- removed `test_batch_read_metadata_multi` as `self._versions.items()`
has versions that have already been deleted, the test tries to lookup
the deleted versions and tries to asserts that they are found
- removed `test_batch_read_snapshot` as the `VersionStoreComparison`
creates a test sequence where it first deletes a symbol, then with
`test_batch_read_snapshot` checks that the versions in the snapshot
match the versions read with `batch_read` which is not the expected
behaviour because deleted versions will not be returned by batch read.
- similarly removed `test_batch_read_tombstoned_version_via_snapshot`
as, with `batch_read(symbol, as_of=[version])` , we shouldn't expect to
read a snapshotted version that has been deleted.


#### Checklist

<details>
  <summary>
   Checklist for code changes...
  </summary>
 
- [ ] Have you updated the relevant docstrings, documentation and
copyright notice?
- [ ] Is this contribution tested against [all ArcticDB's
features](../docs/mkdocs/docs/technical/contributing.md)?
- [ ] Do all exceptions introduced raise appropriate [error
messages](https://docs.arcticdb.io/error_messages/)?
 - [ ] Are API changes highlighted in the PR description?
- [ ] Is the PR labelled as enhancement or bug so it appears in
autogenerated release notes?
</details>

<!--
Thanks for contributing a Pull Request to ArcticDB! Please ensure you
have taken a look at:
- ArcticDB's Code of Conduct:
https://github.com/man-group/ArcticDB/blob/master/CODE_OF_CONDUCT.md
- ArcticDB's Contribution Licensing:
https://github.com/man-group/ArcticDB/blob/master/docs/mkdocs/docs/technical/contributing.md#contribution-licensing
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Bug fixing drive Q1 2024 bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants