Skip to content

[Transform] Specify use_point_in_time in settings #5322

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Oct 9, 2025
Merged

[Transform] Specify use_point_in_time in settings #5322

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
f8f5745
[Transform] Specify use_point_in_time in settings
prwhelan Sep 19, 2025
de6d072
use @ext_doc_id
prwhelan Sep 19, 2025
f03d532
Merge branch 'main' into transform/pit
kosabogi Sep 23, 2025
7765806
Merge branch 'main' into transform/pit
prwhelan Oct 1, 2025
e6f662c
address comments
prwhelan Oct 1, 2025
22bdb38
Merge branch 'main' into transform/pit
prwhelan Oct 9, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 19 additions & 4 deletions output/schema/schema.json
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions output/typescript/types.ts
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 1 addition & 1 deletion specification/_doc_ids/table.csv
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -521,7 +521,7 @@ painless-contexts,https://www.elastic.co/docs/reference/scripting-languages/pain
painless-execute-api,https://www.elastic.co/docs/reference/scripting-languages/painless/painless-api-examples,,
pipeline-processor,https://www.elastic.co/docs/reference/enrich-processor/pipeline-processor,,
pki-realm,https://www.elastic.co/docs/deploy-manage/users-roles/cluster-or-deployment-auth/pki,,
point-in-time-api,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/point-in-time-api.html,
point-in-time-api,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/point-in-time-api.html,Open a point in time API
prevalidate-node-removal,https://www.elastic.co/docs/api/doc/elasticsearch/group/endpoint-cluster,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/cluster.html,
preview-dfanalytics,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-ml-preview-data-frame-analytics,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/preview-dfanalytics.html,
preview-transform,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-transform-preview-transform,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/preview-transform.html,
Expand Down
10 changes: 10 additions & 0 deletions specification/transform/_types/Transform.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,16 @@ export class Settings {
* @server_default 500
*/
max_page_search_size?: integer
/**
* Specifies whether the transform checkpoint will use the Point In Time API while searching over the source index.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What happens if you set it to false? It uses the scroll API?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nah it just uses a search request without PIT: https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/transform/src/main/java/org/elasticsearch/xpack/transform/transforms/TransformIndexer.java#L1154

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ideally we'd explain the pros/cons differences of each approach, or at least outline why you'd want to use the PIT, rather than just the how :)

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should that go on the PIT page? in either case, we'd need someone familiar with PIT to go into that detail, I'm just trying to show what users can configure rather than keeping the setting hidden

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think what I missed is that transforms use composite aggregations, which offer pagination without using PIT. However, transforms use PIT by default, as it returns more correct results. That said, as it can be expensive if many PITs are open at any given time, this can be disabled. Is my understanding correct?

This is not about explaining what PIT does (especially given that we have a link), but making it clear what the setting does. For example, max_page_search_size is clear here!

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not about explaining what PIT does (especially given that we have a link), but making it clear what the setting does.

right Quentin said it much clearer than me 😄

* In general, Point In Time is an optimization that will reduce pressure on the source index by reducing the amount
* of refreshes and merges, but it can be expensive if a large number of Point In Times are opened and closed for a
* given index. The benefits and impact depend on the data being searched, the ingest rate into the source index, and
* the amount of other consumers searching the same source index.
* @ext_doc_id point-in-time-api
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@elastic/developer-docs Is this a correct usage of ext_doc_id?

This Transform API can be configured to use the PIT API, and here point-in-time-api links to the point-in-time API docs.

Copy link
Contributor

@kosabogi kosabogi Sep 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey @pquentin, the @ext_doc_id looks fine here. I see it’s already in the table.csv file as well, so it should be good. What we usually do to make sure the links appear correctly is test them locally by running these (you should have the bump CLI installed):

make generate
make transform-to-openapi
bump preview output/openapi/elasticsearch-openapi.json

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems to have worked as intended:
image
And it correctly links to https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time

Copy link
Contributor

@leemthompo leemthompo Sep 22, 2025 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FYI we could add a description field to the link definition in

elasticsearch-specification/specification/_doc_ids/table.csv

Line 521 in 34dfe8b

point-in-time-api,https://www.elastic.co/docs/api/doc/elasticsearch/operation/operation-open-point-in-time,https://www.elastic.co/guide/en/elasticsearch/reference/8.18/point-in-time-api.html,

and this would be render nicer link text than just External documentation 💅

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you have an example of that?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this could be as simple as Point in Time API

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can see an example in this PR: https://github.com/elastic/elasticsearch-specification/pull/5331/files#diff-afcc28b0721ba019d194536c1ff7cb6117ac7db92b094fbc8723ec371771385d

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that it would be nice to name this "Open a point in time API" instead of "External documentation."

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It didn't seem to work, but perhaps I did it wrong:
image

All the others from that example PR still show External documentation for me so 🤷

* @server_default true
*/
use_point_in_time?: boolean

/**
* If `true`, the transform runs in unattended mode. In unattended mode, the transform retries indefinitely in case
Expand Down
Loading