Object storage listing #4434

SammyVimes · 2024-05-10T07:31:37Z

Changelog category

New feature

Proposed docs

Object Storage Listing API

Objects in (S3-like) object storage have a name column. Although object storage is unstructured, users often give their objects name similar to file path, e.g. /foo/baz.txt, /foo/bar/test.txt. One of the common operations is listing. For example, listing folder /foo/ should yield:

/bar/
/foo/baz.txt

Request

Table name

Name of the table to list objects from

Key prefix

Limits the response to keys that begin with the specified key prefix. keyColumn0 ... path column (exclusive) values.

Example in Go:

Ydb.TypedValue{
    Type: &Ydb.Type{
        Type: &Ydb.Type_TupleType{
            TupleType: &Ydb.TupleType{
                Elements: []*Ydb.Type{
                    {
                        Type: &Ydb.Type_TypeId{
                            TypeId: Ydb.Type_UINT64,
                        },
                    },
                },
            },
        },
    },
    Value: &Ydb.Value{
        Items: []*Ydb.Value{
            {
                Value: &Ydb.Value_Uint64Value{
                    Uint64Value: 100,
                },
            },
        },
    },
}

This is a Ydb.TypedValue that contains key column values in PK order. Must have all columns from first until the "path" column (excluding it).

Path column prefix

Limits the response to keys that begin with the specified path prefix (e.g. /foo)

Path column delimiter

A delimiter is a character that you use to group keys. For example, if path is /foo/bar/baz.txt, then delimiter should be "/"

Start after key suffix

Key's suffix column values: path column (inclusive) ... columnN values.
StartAfterKeySuffix is where you want to start listing from. This API starts listing after this specified key. StartAfterKeySuffix can be any existing or non-existing key

Example in Go:

Ydb.TypedValue{
    Type: &Ydb.Type{
        Type: &Ydb.Type_TupleType{
            TupleType: &Ydb.TupleType{
                Elements: []*Ydb.Type{
                    {
                        Type: &Ydb.Type_TypeId{
                            TypeId: Ydb.Type_UTF8,
                        },
                    },
                },
            },
        },
    },
    Value: &Ydb.Value{
        Items: []*Ydb.Value{
            {
                Value: &Ydb.Value_TextValue{
                    TextValue: "/Foo/bar/",
                },
            },
        },
    },
}

This is a Ydb.TypedValue that contains key column values in PK order.

Matching filter

Column values to filter out rows

Here is a filter that filters out all rows with "deleted" column's value equal to true.
Example in Go:

Ydb.TypedValue{
    Type: &Ydb.Type{
        Type: &Ydb.Type_TupleType{
            TupleType: &Ydb.TupleType{
                Elements: []*Ydb.Type{
                    {
                        Type: &Ydb.Type_ListType{
                            ListType: &Ydb.ListType{
                                Item: &Ydb.Type{
                                    Type: &Ydb.Type_TypeId{
                                        TypeId: Ydb.Type_STRING,
                                    },
                                },
                            },
                        },
                    },
                    {
                        Type: &Ydb.Type_ListType{
                            ListType: &Ydb.ListType{
                                Item: &Ydb.Type{
                                    Type: &Ydb.Type_TypeId{
                                        TypeId: Ydb.Type_UINT32,
                                    },
                                },
                            },
                        },
                    },
                    {
                        Type: &Ydb.Type_TupleType{
                            TupleType: &Ydb.TupleType{
                                Elements: []*Ydb.Type{
                                    {
                                        Type: &Ydb.Type_TypeId{
                                            TypeId: Ydb.Type_BOOL,
                                        },
                                    },
                                },
                            },
                        },
                    },
                },
            },
        },
    },
    Value: &Ydb.Value{
        Items: []*Ydb.Value{
            {
                Items: []*Ydb.Value{
                    {
                        Value: &Ydb.Value_TextValue{
                            TextValue: "deleted",
                        },
                    },
                },
            },
            {
                Items: []*Ydb.Value{
                    {
                        Value: &Ydb.Value_Uint32Value{
                            Uint32Value: uint32(Ydb_ObjectStorage.ListingRequest_EQUAL),
                        },
                    },
                },
            },
            {
                Items: []*Ydb.Value{
                    {
                        Value: &Ydb.Value_BoolValue{
                            BoolValue: true,
                        },
                    },
                },
            },
        },
    },
}

This is a Ydb.TypedValue that contains list of column names, list of matching types (either EQUAL or NOT_EQUAL) and list of values for the columns.

Continuation token

ContinuationToken indicates that the list is being continued on with a token. ContinuationToken is obfuscated and is not a real key. You can use this ContinuationToken for pagination of the list results

Columns to return

Which columns of the row to return. Path column and filter columns are added automatically

Max keys

Sets the maximum number of keys returned in the response. By default, the action returns up to 1,000 key names. The response might contain fewer keys but will never contain more

Response

Status

YDB operation status code

Issues

YDB operation issue messages

Common prefixes

All the keys (up to 1_000) that share the same prefix are grouped together. When counting the total numbers of returns by this API operation, this group of keys is considered as one item.
A response can contain CommonPrefixes only if you specify a delimiter.
CommonPrefixes contains all (if there are any) keys between Prefix and the next occurrence of the string specified by a delimiter.

Next continuation token

NextContinuationToken is sent when isTruncated is true, which means there are more keys that can be listed. The next list requests can be continued with this NextContinuationToken. NextContinuationToken is obfuscated and is not a real key

Is truncated

Set to false if all of the results were returned. Set to true if more keys are available to return

Fixes ydb-platform#3217

github-actions · 2024-05-10T07:34:48Z

⚪ 2024-05-10 07:34:48 UTC Pre-commit check for ae3004d has started.
⚪ 2024-05-10 07:34:50 UTC Build linux-x86_64-release-clang14 is running...
🔴 2024-05-10 07:51:11 UTC Build failed. see the build logs.

github-actions · 2024-05-10T07:34:59Z

⚪ 2024-05-10 07:34:59 UTC Pre-commit check for ae3004d has started.
⚪ 2024-05-10 07:35:01 UTC Build linux-x86_64-relwithdebinfo is running...
🔴 2024-05-10 07:52:58 UTC Build failed. see the build logs.
🔴 2024-05-10 07:54:34 UTC Tests run skipped.

github-actions · 2024-05-10T07:34:59Z

⚪ 2024-05-10 07:34:59 UTC Pre-commit check for ae3004d has started.
⚪ 2024-05-10 07:35:01 UTC Build linux-x86_64-release-asan is running...
🔴 2024-05-10 07:54:26 UTC Build failed. see the build logs.
🔴 2024-05-10 07:56:05 UTC Tests run skipped.

github-actions · 2024-05-10T08:21:53Z

⚪ 2024-05-10 08:21:52 UTC Pre-commit check for 7094dea has started.
⚪ 2024-05-10 08:21:55 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-10 08:23:45 UTC Build successful.

github-actions · 2024-05-10T08:22:09Z

⚪ 2024-05-10 08:22:09 UTC Pre-commit check for 7094dea has started.
⚪ 2024-05-10 08:22:12 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-10 08:24:14 UTC Build successful.
⚪ 2024-05-10 08:26:07 UTC Tests are running...
🔴 2024-05-10 10:22:48 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73530	60315	0	3	13205	7

github-actions · 2024-05-10T08:22:10Z

⚪ 2024-05-10 08:22:09 UTC Pre-commit check for 7094dea has started.
⚪ 2024-05-10 08:22:12 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-10 08:24:14 UTC Build successful.
⚪ 2024-05-10 08:26:08 UTC Tests are running...
🔴 2024-05-10 10:20:34 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14690	14549	0	48	84	9

github-actions · 2024-05-10T12:31:20Z

⚪ 2024-05-10 12:31:19 UTC Pre-commit check for 1cd3eb8 has started.
⚪ 2024-05-10 12:31:22 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-10 12:46:41 UTC Build successful.

github-actions · 2024-05-10T12:31:21Z

⚪ 2024-05-10 12:31:21 UTC Pre-commit check for 1cd3eb8 has started.
⚪ 2024-05-10 12:31:23 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-10 12:48:55 UTC Build successful.
⚪ 2024-05-10 12:50:43 UTC Tests are running...
🔴 2024-05-10 14:42:22 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14690	14541	0	57	84	8

github-actions · 2024-05-10T12:32:02Z

⚪ 2024-05-10 12:32:02 UTC Pre-commit check for 1cd3eb8 has started.
⚪ 2024-05-10 12:32:04 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-10 12:47:55 UTC Build successful.
⚪ 2024-05-10 12:49:42 UTC Tests are running...
🔴 2024-05-10 14:43:47 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73528	60297	0	8	13215	8

github-actions · 2024-05-14T09:24:23Z

⚪ 2024-05-14 09:24:23 UTC Pre-commit check for f8ea568 has started.
⚪ 2024-05-14 09:24:26 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-14 09:41:58 UTC Build successful.
⚪ 2024-05-14 09:43:49 UTC Tests are running...
🔴 2024-05-14 11:29:25 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14667	14564	0	36	58	9

github-actions · 2024-05-14T09:26:04Z

⚪ 2024-05-14 09:26:04 UTC Pre-commit check for f8ea568 has started.
⚪ 2024-05-14 09:26:06 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-14 09:42:30 UTC Build successful.

github-actions · 2024-05-14T09:26:19Z

⚪ 2024-05-14 09:26:19 UTC Pre-commit check for f8ea568 has started.
⚪ 2024-05-14 09:26:22 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-14 09:45:57 UTC Build successful.
⚪ 2024-05-14 09:47:52 UTC Tests are running...
🔴 2024-05-14 11:47:42 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73509	60312	0	2	13188	7

ydb/core/tx/datashard/datashard.h

github-actions · 2024-05-14T21:08:49Z

⚪ 2024-05-14 21:08:49 UTC Pre-commit check for 8395348 has started.
⚪ 2024-05-14 21:08:51 UTC Build linux-x86_64-release-clang14 is running...
🟢 2024-05-14 21:25:27 UTC Build successful.

github-actions · 2024-05-14T21:08:49Z

⚪ 2024-05-14 21:08:49 UTC Pre-commit check for 8395348 has started.
⚪ 2024-05-14 21:08:51 UTC Build linux-x86_64-release-asan is running...
🟢 2024-05-14 21:28:20 UTC Build successful.
⚪ 2024-05-14 21:30:18 UTC Tests are running...
🔴 2024-05-14 23:20:38 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
14681	14565	0	43	59	14

github-actions · 2024-05-14T21:13:46Z

⚪ 2024-05-14 21:13:46 UTC Pre-commit check for 8395348 has started.
⚪ 2024-05-14 21:13:49 UTC Build linux-x86_64-relwithdebinfo is running...
🟢 2024-05-14 21:32:08 UTC Build successful.
⚪ 2024-05-14 21:34:03 UTC Tests are running...
🔴 2024-05-14 23:36:07 UTC Some tests failed, follow the links below.

Test history

TESTS	PASSED	ERRORS	FAILED	SKIPPED	MUTED^?
73526	60324	0	6	13188	8

Fixes ydb-platform#4348

SammyVimes added 3 commits May 9, 2024 21:30

Initial Object Storage Listing internal handler

4c712c9

YDB-3217 Object storage listing filtering (ydb-platform#3215)

d385590

Fixes ydb-platform#3217

Object Listing filter API update (ydb-platform#4318)

a45b8b9

github-actions bot added the new-feature label May 10, 2024

Fix compilation

88b4cbc

.

82c5015

Final polishes

40fcb99

azevaykin reviewed May 14, 2024

View reviewed changes

ydb/core/tx/datashard/datashard.h Show resolved Hide resolved

Add datashard tx test

be928e9

dcherednik approved these changes May 16, 2024

View reviewed changes

SammyVimes merged commit 48a3069 into ydb-platform:main May 17, 2024
3 of 5 checks passed

SammyVimes mentioned this pull request May 21, 2024

Add object storage listing API draft ydb-platform/ydb-api-protos#39

Merged

serbel324 mentioned this pull request May 24, 2024

Add description of new ICB controls for CMS #4818

Merged

This was referenced May 27, 2024

Cli changes for autoscaling #4871

Closed

Fix getting mid of range for autoscaling #4884

Closed

niksaveliev mentioned this pull request May 27, 2024

Fix getting mid range #4885

Merged

MrLolthe1st pushed a commit to MrLolthe1st/ydb that referenced this pull request May 28, 2024

Object storage listing (ydb-platform#4434)

9699097

Fixes ydb-platform#4348

niksaveliev mentioned this pull request May 29, 2024

Autoscaling logs #4976

Merged

This was referenced May 29, 2024

YQ-3292 fixed kqprun ResourceBrokerConfig passing #4994

Merged

Added ExecutionId for script execution logs #5014

Merged

This was referenced May 30, 2024

Forbid autoscaling with size retention #5054

Merged

Mute split by load ut #5264

Closed

Mute split by load ut #5267

Closed

Mute split by load ut #5272

Merged

This was referenced Jun 7, 2024

Fix build #5310

Merged

Fix split by load ut #5383

Merged

niksaveliev mentioned this pull request Jun 17, 2024

Fix win build #5622

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Object storage listing #4434

Object storage listing #4434

SammyVimes commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

Object storage listing #4434

Object storage listing #4434

Conversation

SammyVimes commented May 10, 2024 • edited Loading

Changelog category

Proposed docs

Object Storage Listing API

Request

Table name

Key prefix

Path column prefix

Path column delimiter

Start after key suffix

Matching filter

Continuation token

Columns to return

Max keys

Response

Status

Issues

Common prefixes

Contents

Next continuation token

Is truncated

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 10, 2024 • edited Loading

github-actions bot commented May 14, 2024 • edited Loading

github-actions bot commented May 14, 2024 • edited Loading

github-actions bot commented May 14, 2024 • edited Loading

github-actions bot commented May 14, 2024 • edited Loading

github-actions bot commented May 14, 2024 • edited Loading

github-actions bot commented May 14, 2024 • edited Loading

SammyVimes commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 10, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading

github-actions bot commented May 14, 2024 •

edited

Loading