Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CARBONDATA-3273] [CARBONDATA-3274] Fix for SORT_SCOPE in CarbonLoadDataCommand #3103

Closed
wants to merge 1 commit into from
Closed

[CARBONDATA-3273] [CARBONDATA-3274] Fix for SORT_SCOPE in CarbonLoadDataCommand #3103

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/configuration-parameters.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ RESET
| carbon.options.date.format | Specifies the data format of the date columns in the data being loaded |
| carbon.options.timestamp.format | Specifies the timestamp format of the time stamp columns in the data being loaded |
| carbon.options.sort.scope | Specifies how the current data load should be sorted with. **NOTE:** Refer to [Data Loading Configuration](#data-loading-configuration)#carbon.sort.scope for detailed information. |
| carbon.table.load.sort.scope | Overrides the SORT_SCOPE provided in CREATE TABLE. |
| carbon.table.load.sort.scope.<db>.<table> | Overrides the SORT_SCOPE provided in CREATE TABLE. |
| carbon.options.global.sort.partitions | |
| carbon.options.serialization.null.format | Default Null value representation in the data being loaded. **NOTE:** Refer to [Data Loading Configuration](#data-loading-configuration)#carbon.options.serialization.null.format for detailed information. |
| carbon.query.directQueryOnDataMap.enabled | Specifies whether datamap can be queried directly. This is useful for debugging purposes.**NOTE: **Refer to [Query Configuration](#query-configuration) for detailed information. |
Expand Down
18 changes: 13 additions & 5 deletions docs/dml-of-carbondata.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,11 +109,19 @@ CarbonData DML statements are documented here,which includes:
```

- ##### SORT_SCOPE:
Sort Scope to be used for the current load. This overrides the Sort Scope of Table.

```
OPTIONS('SORT_SCOPE'='BATCH_SORT')
```
Sort Scope to be used for the current load. This overrides the Sort Scope of Table.
Requirement: Sort Columns must be set while creating table. If Sort Columns is null, Sort Scope is always NO_SORT.

```
OPTIONS('SORT_SCOPE'='BATCH_SORT')
```

Priority order for choosing Sort Scope is:
1. Load Data Command
2. CARBON.TABLE.LOAD.SORT.SCOPE.<db>.<table> session property
3. Table level Sort Scope
4. CARBON.OPTIONS.SORT.SCOPE session property
5. Default Value: NO_SORT

- ##### MULTILINE:

Expand Down
28 changes: 14 additions & 14 deletions .../main/scala/org/apache/spark/sql/execution/command/management/CarbonLoadDataCommand.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -200,26 +200,26 @@ case class CarbonLoadDataCommand(
* LOAD DATA INPATH 'data.csv' INTO TABLE tableName OPTIONS('sort_scope'='no_sort')
*
* 2. Session property CARBON_TABLE_LOAD_SORT_SCOPE ->
* SET CARBON.TABLE.LOAD.SORT.SCOPE.database.table=no_sort
* SET CARBON.TABLE.LOAD.SORT.SCOPE.database.table=batch_sort
* SET CARBON.TABLE.LOAD.SORT.SCOPE.database.table=local_sort
* SET CARBON.TABLE.LOAD.SORT.SCOPE.database.table=global_sort
*
* 3. Sort Scope provided in TBLPROPERTIES
* 4. Session property CARBON_OPTIONS_SORT_SCOPE
* 5. Default Sort Scope LOAD_SORT_SCOPE
*/
if (tableProperties.get(CarbonCommonConstants.SORT_COLUMNS) != null &&
tableProperties.get(CarbonCommonConstants.SORT_SCOPE) == null) {
// If there are Sort Columns given for the table and Sort Scope is not specified,
// we will take it as whichever sort scope given or LOCAL_SORT as default
optionsFinal
.put(CarbonCommonConstants.SORT_SCOPE,
carbonProperty
.getProperty(
CarbonLoadOptionConstants.CARBON_TABLE_LOAD_SORT_SCOPE + table.getDatabaseName + "." +
table.getTableName, carbonProperty.getProperty(CarbonCommonConstants.LOAD_SORT_SCOPE,
SortScopeOptions.getSortScope("LOCAL_SORT").toString)))
if (StringUtils.isBlank(tableProperties.get(CarbonCommonConstants.SORT_COLUMNS))) {
// If tableProperties.SORT_COLUMNS is null
optionsFinal.put(CarbonCommonConstants.SORT_SCOPE,
SortScopeOptions.SortScope.NO_SORT.name)
} else if (StringUtils.isBlank(tableProperties.get(CarbonCommonConstants.SORT_SCOPE))) {
// If tableProperties.SORT_COLUMNS is not null
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please optimize the comment here:
in case SORT_COLUMNS is not set and SORT_SCOPE is set

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment

    else if (StringUtils.isBlank(tableProperties.get(CarbonCommonConstants.SORT_SCOPE))) {
      // If tableProperties.SORT_COLUMNS is not null
      // and tableProperties.SORT_SCOPE is null
      optionsFinal.put(CarbonCommonConstants.SORT_SCOPE,

is already there.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By saying ‘set’ or 'not set', it refers to the business logic while ‘null’ or 'not null' does not refer to that logic directly.

// and tableProperties.SORT_SCOPE is null
optionsFinal.put(CarbonCommonConstants.SORT_SCOPE,
options.getOrElse(CarbonCommonConstants.SORT_SCOPE,
carbonProperty.getProperty(CarbonLoadOptionConstants.CARBON_TABLE_LOAD_SORT_SCOPE +
table.getDatabaseName + "." + table.getTableName,
carbonProperty.getProperty(CarbonLoadOptionConstants.CARBON_OPTIONS_SORT_SCOPE,
carbonProperty.getProperty(CarbonCommonConstants.LOAD_SORT_SCOPE,
SortScopeOptions.SortScope.LOCAL_SORT.name)))))
} else {
optionsFinal.put(CarbonCommonConstants.SORT_SCOPE,
options.getOrElse(CarbonCommonConstants.SORT_SCOPE,
Expand Down