[BUG] Paimon catalog scan phase always assigin 3 backends

[yohengyang]

https://forum.mirrorship.cn/t/topic/12192/5

问题描述

查询一个paimon 大表非常慢，表总数据行数大约22 亿，集群 100 个 backends节点，查看 profile发现数据 scan只有 3 实例运行

Starrocks版本：社区3.2.4
paimon 表 有200 bucket
查询语句：select count(1) from paimon.xxx.xxxx;

定位到问题原因

com.starrocks.qe.HDFSBackendSelector.HdfsScanRangeHasher#acceptScanRangeLocations没有考虑到paimon

下面是一个 paimon的THdfsScanRange具体信息

没有一个条件命中com.starrocks.qe.HDFSBackendSelector.HdfsScanRangeHasher#acceptScanRangeLocations中的primitiveSink，所以最终所有 paimon THdfsScanRange计算的 hash 值都是 0

starrocks/fe/fe-core/src/main/java/com/starrocks/qe/HDFSBackendSelector.java

Lines 127 to 145 in 613f0b5

	public void acceptScanRangeLocations(TScanRangeLocations tScanRangeLocations, PrimitiveSink primitiveSink) {
	THdfsScanRange hdfsScanRange = tScanRangeLocations.scan_range.hdfs_scan_range;
	if (hdfsScanRange.isSetFull_path()) {
	primitiveSink.putString(hdfsScanRange.full_path, StandardCharsets.UTF_8);
	} else {
	if (hdfsScanRange.isSetPartition_id() &&
	predicates.getIdToPartitionKey().containsKey(hdfsScanRange.getPartition_id())) {
	PartitionKey partitionKey = predicates.getIdToPartitionKey().get(hdfsScanRange.getPartition_id());
	primitiveSink.putInt(partitionKey.hashCode());
	}
	if (hdfsScanRange.isSetRelative_path()) {
	primitiveSink.putString(hdfsScanRange.relative_path, StandardCharsets.UTF_8);
	}
	}
	if (hdfsScanRange.isSetOffset()) {
	primitiveSink.putLong(hdfsScanRange.getOffset());
	}
	}
	}

因为 hash值都是 0， 所以 com.starrocks.qe.HDFSBackendSelector#computeScanRangeAssignment中hashring 返回的始终是同一组 backend

starrocks/fe/fe-core/src/main/java/com/starrocks/qe/HDFSBackendSelector.java

Lines 278 to 279 in 613f0b5

	List<ComputeNode> backends = hashRing.get(scanRangeLocations, kCandidateNumber);
	ComputeNode node = reBalanceScanRangeForComputeNode(backends, avgNodeScanRangeBytes, scanRangeLocations);

一个修复思路是在PaimonScanNode中把 split bucket 信息放入THdfsScanRange，HDFSBackendSelector根据 bucket 信息做 hash

starrocks/fe/fe-core/src/main/java/com/starrocks/planner/PaimonScanNode.java

Lines 208 to 226 in 613f0b5

	private void addSplitScanRangeLocations(Split split, String predicateInfo, long totalFileLength) {
	TScanRangeLocations scanRangeLocations = new TScanRangeLocations();
	THdfsScanRange hdfsScanRange = new THdfsScanRange();
	hdfsScanRange.setUse_paimon_jni_reader(true);
	hdfsScanRange.setPaimon_split_info(encodeObjectToString(split));
	hdfsScanRange.setPaimon_predicate_info(predicateInfo);
	hdfsScanRange.setFile_length(totalFileLength);
	hdfsScanRange.setLength(totalFileLength);
	TScanRange scanRange = new TScanRange();
	scanRange.setHdfs_scan_range(hdfsScanRange);
	scanRangeLocations.setScan_range(scanRange);
	TScanRangeLocation scanRangeLocation = new TScanRangeLocation(new TNetworkAddress("-1", -1));
	scanRangeLocations.addToLocations(scanRangeLocation);
	scanRangeLocationsList.add(scanRangeLocations);
	}

[miomiocat]

@yohengyang 感谢反馈，请问下，你们的这个case里，具体有多少个remoteScanRangeLocations(PaimonScanNode::getScanRangeLocations()的返回值)？

[yohengyang]

@yohengyang 感谢反馈，请问下，你们的这个case里，具体有多少个remoteScanRangeLocations(PaimonScanNode::getScanRangeLocations()的返回值)？

这个 case有 200个TScanRangeLocations，我们的表是 200bucket