-
Notifications
You must be signed in to change notification settings - Fork 467
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add object storage #213
Add object storage #213
Conversation
Codecov Report
Additional details and impacted files@@ Coverage Diff @@
## dev #213 +/- ##
==========================================
- Coverage 92.89% 91.37% -1.52%
==========================================
Files 42 47 +5
Lines 1365 1496 +131
==========================================
+ Hits 1268 1367 +99
- Misses 97 129 +32
|
0face8e
to
328396e
Compare
gptcache/adapter/adapter.py
Outdated
"question": cache_question, | ||
"answer": cache_answer, | ||
"question": ret.question, | ||
"answer": ret.answer[0].answer, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
answers
is more suitable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
gptcache/adapter/adapter.py
Outdated
}, | ||
extra_param=context.get("evaluation_func", None), | ||
) | ||
if rank_threshold <= rank: | ||
cache_answers.append((rank, cache_answer)) | ||
cache_answers.append((rank, ret.answer[0].answer)) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
ditto
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
gptcache/manager/data_manager.py
Outdated
@@ -180,10 +188,16 @@ def import_data( | |||
normalize(embedding_data) for embedding_data in embedding_datas | |||
] | |||
for i, embedding_data in enumerate(embedding_datas): | |||
if not isinstance(answers[i], str): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
maybe is a list, one question multiple answers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
gptcache/manager/data_manager.py
Outdated
self.import_data([question], [answer], [embedding_data]) | ||
|
||
def _process_answers(self, answers: Union[Answer, List[Answer]]): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
_process_answers -> _process_answer_data
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
"""Local object storage | ||
""" | ||
|
||
def __init__(self, local_root): |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
add param type, like: local_root: str
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
try: | ||
os.remove(obj) | ||
except Exception: # pylint: disable=broad-except | ||
pass |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
give some log
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
def get(name, **kwargs): | ||
if name == "local": | ||
from gptcache.manager.object_data.local_storage import LocalObjectStorage # pylint: disable=import-outside-toplevel | ||
object_base = LocalObjectStorage(kwargs.get("path", "./")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the default value ./local_obj
may be more suitable
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
elif name == "s3": | ||
from gptcache.manager.object_data.s3_storage import S3Storage # pylint: disable=import-outside-toplevel | ||
object_base = S3Storage(kwargs.get("path_prefix"), kwargs.get("bucket"), | ||
kwargs.get("access_key"), kwargs.get("secret_key")) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
need endpoint
param
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
gptcache/manager/scalar_data/base.py
Outdated
question: Any | ||
answer: Union[Any, List[Any]] | ||
answer: List[Answer] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
rename it, answers
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
fixed
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
/lgtm |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: junjiejiangjjj, SimFG The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Signed-off-by: junjie.jiang <junjie.jiang@zilliz.com>
No description provided.