Discuss use of partitioning on S3 page

[DanRoscigno]

@qoega I did not remove the partition by as we mention using the partitions when parallelizing writes to s3:

Note the use of partitioning on the pickup_date field. Whilst usually a technique to assist with data management, we can later use this key to parallelize writes to s3.

Can you please pop a note in here and we can discuss whether or not to make a change?

[qoega]

I will try ask developers more about + and - of using partition in Cloud.
As I remember the only reason to use them was possibility to easily drop them or replace with other prepared partition. Just want to have more confidence.

we can later use this key to parallelize writes to s3.

I never checked the speed if we use PARTITION BY computed column instead of using partition key to write to S3.

CREATE TABLE visits_with_partitions
(
    VisitDate Date,
    Hour UInt8,
    ClientID UUID
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(VisitDate)
ORDER BY Hour;

CREATE TABLE visits_without_partitions
(
    VisitDate Date,
    Hour UInt8,
    ClientID UUID
)
ENGINE = MergeTree()
ORDER BY Date, Hour;

To check something like

INSERT INTO TABLE FUNCTION
    s3('http://bucket.amazonaws.com/my_bucket/visits_{_partition_id}.csv', 'CSV')
    PARTITION BY Date
SELECT * FROM visits_with_partitions;
INSERT INTO TABLE FUNCTION
    s3('http://bucket.amazonaws.com/my_bucket/visits_{_partition_id}.csv', 'CSV')
    PARTITION BY Date
SELECT * FROM visits_without_partitions;
INSERT INTO TABLE FUNCTION
    s3('http://bucket.amazonaws.com/my_bucket/visits_{_partition_id}.csv', 'CSV')
    PARTITION BY (Date::String || Hour::Date)
SELECT * FROM visits_without_partitions;

[DanRoscigno]

Note is in place:

note

In most cases you do not need a partition key, and in most other cases you do not need a partition key more granular than by months.

You should never use too granular of partitioning. Don't partition your data by client identifiers or names. Instead, make a client identifier or name the first column in the ORDER BY expression.