using the web UI to estimate time remaining for query exection?

[gideonshalev]

Is it possible to look at the web UI for a given query ID and estimate how much time is left until the query completes?
For example, in the query details page, at the very bottom, there's information about each task. But even looking at the number of completed splits, it's hard to tell how much is left as I can't tell how many total splits to expect.
Is there some way of figuring this out in order to get a better estimate of query progress?

[gideonshalev]

@martint - is this the right place to ask this question?

[hashhar]

Hi @gideonshalev. Sorry it took so long to see this - but it seems your question has an answer here -> https://trinodb.slack.com/archives/CFLB9AMBN/p1621456437030100

TL; DR: Progress is calculated as completedSplits / totalSplits. Note that the totalSplits value is not known until scheduling is complete. So the UI waits until scheduled = true before actually showing progress.

As for estimating the time remaning - I'm not sure there's any way to do it accurately because each split might take a different amount of time, a query may take shorter or longer across different executions due to concurrent queries etc. The best you can estimate is query progress as a percentage of splits completed. Another option is to keep track of historical query durations and use them to guess the expected duration.

[gideonshalev]

@hashhar thx for the info! But I still don't see how to introspect the scheduling progress. Where in the UI can you see that? Or if not in the UI, can it be gotten at in some other way?
Even if we can't predict the time each split will take to complete, we should be able to calculate the total number of splits required pretty early in the process, right?

[hashhar]

@gideonshalev Total number of splits can change while the query is being scheduled because the query starts executing as soon as enough splits are created - we don't wait for all splits to get generated. However the actual duration should be a small fraction of the overall query wall-time.

As for how to get the information - if you are using the JDBC driver you can poll ResultSet#getStats which has totalSplits, completedSplits and scheduled flag.

For the Python client you can access the same information via Cursor#stats.

The thread I linked to talks about how to do it in the Python client.
For reference you can also consult the trino-cli source code which prints the query progress -

trino/client/trino-cli/src/main/java/io/trino/cli/StatusPrinter.java

Lines 322 to 353 in 775e856

	if (stats.isScheduled()) {
	String progressBar = formatProgressBar(progressWidth,
	stats.getCompletedSplits(),
	max(0, stats.getRunningSplits()),
	stats.getTotalSplits());
	// 0:17 [ 103MB, 802K rows] [5.74MB/s, 44.9K rows/s] [=====>> ] 10%
	String progressLine = format("%s [%5s rows, %6s] [%5s rows/s, %8s] [%s] %d%%",
	formatTime(wallTime),
	formatCount(stats.getProcessedRows()),
	formatDataSize(bytes(stats.getProcessedBytes()), true),
	formatCountRate(stats.getProcessedRows(), wallTime, false),
	formatDataRate(bytes(stats.getProcessedBytes()), wallTime, true),
	progressBar,
	progressPercentage);
	reprintLine(progressLine);
	}
	else {
	String progressBar = formatProgressBar(progressWidth, Ints.saturatedCast(nanosSince(start).roundTo(SECONDS)));
	// 0:17 [ 103MB, 802K rows] [5.74MB/s, 44.9K rows/s] [ <=> ]
	String progressLine = format("%s [%5s rows, %6s] [%5s rows/s, %8s] [%s]",
	formatTime(wallTime),
	formatCount(stats.getProcessedRows()),
	formatDataSize(bytes(stats.getProcessedBytes()), true),
	formatCountRate(stats.getProcessedRows(), wallTime, false),
	formatDataRate(bytes(stats.getProcessedBytes()), wallTime, true),
	progressBar);
	reprintLine(progressLine);
	}

. This is the relevant section.

[gideonshalev]

@hashhar - awesome - thx!