new upload procedure #789
new upload procedure #789
Conversation
pkasprzyk
force-pushed
the
pk/new-upload
branch
2 times, most recently
from
15f5be2
to
2e55cbd
Compare
pkasprzyk
force-pushed
the
pk/new-upload
branch
from
cd691df
to
69de0c5
Compare
| @@ -15,6 +15,16 @@ | |||
| # | |||
|
|
|||
|
|
|||
| class MultipartConfig: | |||
There was a problem hiding this comment.
It's Legacy Client so I didn't want to introduce dataclasses there. But I'm not sure why 🤣
Done.
|
|
||
| from future.builtins import object | ||
|
|
||
| from neptune.internal.storage.storage_utils import ( | ||
| UploadEntry, | ||
| AttributeUploadConfiguration, | ||
| ) | ||
| from neptune.new.exceptions import InternalClientError | ||
| from neptune.new.internal.backends.api_model import MultipartConfig | ||
|
|
||
|
|
||
| class FileChunk(object): |
There was a problem hiding this comment.
It might be a dataclass as well.
There was a problem hiding this comment.
not really my code, but you're right :)
There was a problem hiding this comment.
You touched this, it's on you :p
| self._max_chunk_size = multipart_config.max_chunk_size | ||
| self._max_chunk_count = multipart_config.max_chunk_count | ||
|
|
||
| def _get_chunk_size(self) -> int: |
There was a problem hiding this comment.
Can we add info here that our aim is to maximize number of chunks?
There was a problem hiding this comment.
Is it, actually? That's just my implementation decision ;)
| if isinstance(chunk, str): | ||
| chunk = chunk.encode("utf-8") | ||
| new_offset = last_offset + len(chunk) | ||
| yield FileChunk(chunk, last_offset, new_offset) |
| def upload_file_attribute( | ||
| swagger_client: SwaggerClient, | ||
| container_id: str, | ||
| attribute: str, | ||
| source: Union[str, bytes], | ||
| ext: str, | ||
| ) -> Optional[NeptuneException]: | ||
| multipart_config: Optional[MultipartConfig], | ||
| ) -> List[NeptuneException]: |
There was a problem hiding this comment.
Is there a situation when we send more than one exception?
There was a problem hiding this comment.
Or we just want to make it consistent with upload_file_set_attribute?
There was a problem hiding this comment.
exactly that, I wanted consistency
No description provided.