Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Allow dynamic_templates but disallow other dynamic mappings #11276

Open
esamattis opened this issue Nov 20, 2023 · 1 comment
Open
Labels
enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing v2.16.0 Issues and PRs related to version 2.16.0 v3.0.0 Issues and PRs related to version 3.0.0

Comments

@esamattis
Copy link

esamattis commented Nov 20, 2023

Is your feature request related to a problem? Please describe.

Currently it is possible to set dynamic: "strict" which prevents adding fields that do not have a static mapping in the "properties" but it also disables usage of dynamic_templates.

For example

{
    "dynamic": "strict",
    "properties": {
        "url": { "type": "keyword" },
    },
    "dynamic_templates": [
        {
            "custom_dates": {
                "match": "date__*",
                "mapping": {
                    "type": "date",
                },
            },
        },
    ],
}

This will disallow documents like

{ "bad": "something" }

while allowing

{ "url": "https://example" }

but it does not allow

{ "date__created": "2023-11-20T16:11:20.598Z" }

which is unfortunate 😞

Describe the solution you'd like

Option like "dynamic": "templates" which allows the date__created and url but disallows the bad field would be really useful to avoid accidental data in the index.

Describe alternatives you've considered

There's a hacky workaround. One can map any remaining fields to a "broken mapping"

{
    "properties": {
        "url": { "type": "keyword" },
    },
    "dynamic_templates": [
        {
            "dates": {
                "match": "date__*",
                "mapping": {
                    "type": "date",
                },
            },
        },
        {
            "reject_rest": {
                "match": "*",
                "mapping": {
                    // Something invalid according to opensearch
                    "__NOT_ALLOWED__": "__NOT_ALLOWED__"
                },
            },
        },
    ],
}

And this works like described but emits bit weird error messages.

unknown parameter [__NOT_ALLOWED__] on mapper [bad] of type [null]
@esamattis esamattis added enhancement Enhancement or improvement to existing feature or request untriaged labels Nov 20, 2023
@esamattis esamattis changed the title [FEATURE] Allow dynamic_templates but disallow other unmapped fields [FEATURE] Allow dynamic_templates but disallow other dynamic mappings Nov 20, 2023
@mch2 mch2 added the Indexing Indexing, Bulk Indexing and anything related to indexing label Dec 8, 2023
@peternied
Copy link
Member

[Triage - attendees 1 2 3 4]
@esamattis Thanks for filing, we'd welcome a pull request for this feature.

@reta reta added v3.0.0 Issues and PRs related to version 3.0.0 v2.16.0 Issues and PRs related to version 2.16.0 labels Jul 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request Indexing Indexing, Bulk Indexing and anything related to indexing v2.16.0 Issues and PRs related to version 2.16.0 v3.0.0 Issues and PRs related to version 3.0.0
Projects
None yet
Development

No branches or pull requests

4 participants