Skip to content

[Filestream] Remove duplicated ECS mappings from package template#17701

Merged
belimawr merged 5 commits intoelastic:mainfrom
belimawr:15704-remove-duplicated-ecs-mappings
Mar 12, 2026
Merged

[Filestream] Remove duplicated ECS mappings from package template#17701
belimawr merged 5 commits intoelastic:mainfrom
belimawr:15704-remove-duplicated-ecs-mappings

Conversation

@belimawr
Copy link
Contributor

@belimawr belimawr commented Mar 6, 2026

Proposed commit message

Remove the imported ECS mappings from the Filestream integration and rely on the ecs@mappings added by Fleet

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.
  • I have verified that any added dashboard complies with Kibana's Dashboard good practices

## Author's Checklist

How to test this PR locally

1.Build the package and start the stack

% cd packages/filestream 
% elastic-package build
elastic-package stack up -v -d --version=9.4.0-SNAPSHOT

2. Add a new Filestream integration

Add a new Filestream integration accepting all of the defaults. Skip installing Elastic Agent
Shortcut: https://localhost:5601/app/fleet/integrations/filestream-2.3.3/add-integration

3. Validate the integration index template

  • Go to Stack Management -> Index Management -> Component Templates
  • Search for filestream, open logs-filestream.generic@package
  • Click on Mappings, you should see the following mappings
logs-filestream.generic@package mappings

{
  "properties": {
    "input": {
      "properties": {
        "type": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "@timestamp": {
      "ignore_malformed": false,
      "type": "date"
    },
    "ecs": {
      "properties": {
        "version": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "log": {
      "properties": {
        "file": {
          "properties": {
            "inode": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "device_id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "fingerprint": {
              "index": false,
              "type": "keyword"
            }
          }
        },
        "offset": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "level": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "data_stream": {
      "properties": {
        "namespace": {
          "type": "constant_keyword"
        },
        "type": {
          "type": "constant_keyword"
        },
        "dataset": {
          "type": "constant_keyword"
        }
      }
    },
    "event": {
      "properties": {
        "original": {
          "index": false,
          "type": "keyword",
          "doc_values": false
        },
        "module": {
          "type": "constant_keyword",
          "value": "filestream"
        },
        "dataset": {
          "type": "constant_keyword"
        }
      }
    },
    "message": {
      "type": "match_only_text"
    },
    "tags": {
      "ignore_above": 1024,
      "type": "keyword"
    }
  }
}

Shortcut: https://localhost:5601/app/management/data/index_management/component_templates/logs-filestream.generic%40package (you still have to click on 'Mappings').

4. Ensure there are no ECS fields defined

Before this PR the mappings were:

mappings before the PR

{
  "dynamic_templates": [
    {
      "_embedded_ecs-ecs_timestamp": {
        "path_match": "@timestamp",
        "mapping": {
          "ignore_malformed": false,
          "type": "date"
        }
      }
    },
    {
      "_embedded_ecs-data_stream_to_constant": {
        "path_match": "data_stream.*",
        "mapping": {
          "type": "constant_keyword"
        }
      }
    },
    {
      "_embedded_ecs-resolved_ip_to_ip": {
        "mapping": {
          "type": "ip"
        },
        "match": "resolved_ip"
      }
    },
    {
      "_embedded_ecs-forwarded_ip_to_ip": {
        "mapping": {
          "type": "ip"
        },
        "match_mapping_type": "string",
        "match": "forwarded_ip"
      }
    },
    {
      "_embedded_ecs-ip_to_ip": {
        "mapping": {
          "type": "ip"
        },
        "match_mapping_type": "string",
        "match": "ip"
      }
    },
    {
      "_embedded_ecs-x509_public_key_exponent_non_indexed_long": {
        "path_match": "*.x509.public_key_exponent",
        "mapping": {
          "index": false,
          "type": "long",
          "doc_values": false
        }
      }
    },
    {
      "_embedded_ecs-port_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "port"
      }
    },
    {
      "_embedded_ecs-thread_id_to_long": {
        "path_match": "*.thread.id",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "_embedded_ecs-status_code_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "status_code"
      }
    },
    {
      "_embedded_ecs-line_to_long": {
        "path_match": "*.file.line",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "_embedded_ecs-priority_to_long": {
        "path_match": "log.syslog.priority",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "_embedded_ecs-code_to_long": {
        "path_match": "*.facility.code",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "_embedded_ecs-bytes_to_long": {
        "mapping": {
          "type": "long"
        },
        "path_unmatch": "*.data.bytes",
        "match": "bytes"
      }
    },
    {
      "_embedded_ecs-packets_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "packets"
      }
    },
    {
      "_embedded_ecs-public_key_exponent_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "public_key_exponent"
      }
    },
    {
      "_embedded_ecs-severity_to_long": {
        "path_match": "event.severity",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "_embedded_ecs-duration_to_long": {
        "path_match": "event.duration",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "_embedded_ecs-pid_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "pid"
      }
    },
    {
      "_embedded_ecs-uptime_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "uptime"
      }
    },
    {
      "_embedded_ecs-sequence_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "sequence"
      }
    },
    {
      "_embedded_ecs-entropy_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "*entropy"
      }
    },
    {
      "_embedded_ecs-size_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "*size"
      }
    },
    {
      "_embedded_ecs-entrypoint_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "entrypoint"
      }
    },
    {
      "_embedded_ecs-ttl_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "ttl"
      }
    },
    {
      "_embedded_ecs-major_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "major"
      }
    },
    {
      "_embedded_ecs-minor_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "minor"
      }
    },
    {
      "_embedded_ecs-as_number_to_long": {
        "path_match": "*.as.number",
        "mapping": {
          "type": "long"
        }
      }
    },
    {
      "_embedded_ecs-pgid_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "pgid"
      }
    },
    {
      "_embedded_ecs-exit_code_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "exit_code"
      }
    },
    {
      "_embedded_ecs-chi_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "chi2"
      }
    },
    {
      "_embedded_ecs-args_count_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "args_count"
      }
    },
    {
      "_embedded_ecs-virtual_address_to_long": {
        "mapping": {
          "type": "long"
        },
        "match": "virtual_address"
      }
    },
    {
      "_embedded_ecs-io_text_to_wildcard": {
        "path_match": "*.io.text",
        "mapping": {
          "type": "wildcard"
        }
      }
    },
    {
      "_embedded_ecs-strings_to_wildcard": {
        "path_match": "registry.data.strings",
        "mapping": {
          "type": "wildcard"
        }
      }
    },
    {
      "_embedded_ecs-path_to_wildcard": {
        "path_match": "*url.path",
        "mapping": {
          "type": "wildcard"
        }
      }
    },
    {
      "_embedded_ecs-message_id_to_wildcard": {
        "mapping": {
          "type": "wildcard"
        },
        "match": "message_id"
      }
    },
    {
      "_embedded_ecs-command_line_to_multifield": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        },
        "match": "command_line"
      }
    },
    {
      "_embedded_ecs-error_stack_trace_to_multifield": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        },
        "match": "stack_trace"
      }
    },
    {
      "_embedded_ecs-http_content_to_multifield": {
        "path_match": "*.body.content",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        }
      }
    },
    {
      "_embedded_ecs-url_full_to_multifield": {
        "path_match": "*url.full",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        }
      }
    },
    {
      "_embedded_ecs-url_original_to_multifield": {
        "path_match": "*url.original",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        }
      }
    },
    {
      "_embedded_ecs-user_agent_original_to_multifield": {
        "path_match": "user_agent.original",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "wildcard"
        }
      }
    },
    {
      "_embedded_ecs-error_message_to_match_only": {
        "path_match": "error.message",
        "mapping": {
          "type": "match_only_text"
        }
      }
    },
    {
      "_embedded_ecs-message_match_only_text": {
        "path_match": "message",
        "mapping": {
          "type": "match_only_text"
        }
      }
    },
    {
      "_embedded_ecs-event_original_non_indexed_keyword": {
        "path_match": "event.original",
        "mapping": {
          "index": false,
          "type": "keyword",
          "doc_values": false
        }
      }
    },
    {
      "_embedded_ecs-agent_name_to_keyword": {
        "path_match": "agent.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-service_name_to_keyword": {
        "path_match": "*.service.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-sections_name_to_keyword": {
        "path_match": "*.sections.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-resource_name_to_keyword": {
        "path_match": "*.resource.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-observer_name_to_keyword": {
        "path_match": "observer.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-question_name_to_keyword": {
        "path_match": "*.question.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-group_name_to_keyword": {
        "path_match": "*.group.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-geo_name_to_keyword": {
        "path_match": "*.geo.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-host_name_to_keyword": {
        "path_match": "host.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-severity_name_to_keyword": {
        "path_match": "*.severity.name",
        "mapping": {
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-title_to_multifield": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        },
        "match": "title"
      }
    },
    {
      "_embedded_ecs-executable_to_multifield": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        },
        "match": "executable"
      }
    },
    {
      "_embedded_ecs-file_path_to_multifield": {
        "path_match": "*.file.path",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-file_target_path_to_multifield": {
        "path_match": "*.file.target_path",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-name_to_multifield": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        },
        "match": "name"
      }
    },
    {
      "_embedded_ecs-full_name_to_multifield": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        },
        "match": "full_name"
      }
    },
    {
      "_embedded_ecs-os_full_to_multifield": {
        "path_match": "*.os.full",
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        }
      }
    },
    {
      "_embedded_ecs-working_directory_to_multifield": {
        "mapping": {
          "fields": {
            "text": {
              "type": "match_only_text"
            }
          },
          "type": "keyword"
        },
        "match": "working_directory"
      }
    },
    {
      "_embedded_ecs-timestamp_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "timestamp"
      }
    },
    {
      "_embedded_ecs-delivery_timestamp_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "delivery_timestamp"
      }
    },
    {
      "_embedded_ecs-not_after_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "not_after"
      }
    },
    {
      "_embedded_ecs-not_before_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "not_before"
      }
    },
    {
      "_embedded_ecs-accessed_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "accessed"
      }
    },
    {
      "_embedded_ecs-origination_timestamp_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "origination_timestamp"
      }
    },
    {
      "_embedded_ecs-created_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "created"
      }
    },
    {
      "_embedded_ecs-installed_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "installed"
      }
    },
    {
      "_embedded_ecs-creation_date_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "creation_date"
      }
    },
    {
      "_embedded_ecs-ctime_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "ctime"
      }
    },
    {
      "_embedded_ecs-mtime_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "mtime"
      }
    },
    {
      "_embedded_ecs-ingested_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "ingested"
      }
    },
    {
      "_embedded_ecs-start_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "start"
      }
    },
    {
      "_embedded_ecs-end_to_date": {
        "mapping": {
          "type": "date"
        },
        "match": "end"
      }
    },
    {
      "_embedded_ecs-score_base_to_float": {
        "path_match": "*.score.base",
        "mapping": {
          "type": "float"
        }
      }
    },
    {
      "_embedded_ecs-score_temporal_to_float": {
        "path_match": "*.score.temporal",
        "mapping": {
          "type": "float"
        }
      }
    },
    {
      "_embedded_ecs-score_to_float": {
        "mapping": {
          "type": "float"
        },
        "match": "*_score"
      }
    },
    {
      "_embedded_ecs-score_norm_to_float": {
        "mapping": {
          "type": "float"
        },
        "match": "*_score_norm"
      }
    },
    {
      "_embedded_ecs-usage_to_float": {
        "mapping": {
          "scaling_factor": 1000,
          "type": "scaled_float"
        },
        "match": "usage"
      }
    },
    {
      "_embedded_ecs-location_to_geo_point": {
        "mapping": {
          "type": "geo_point"
        },
        "match": "location"
      }
    },
    {
      "_embedded_ecs-same_as_process_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "same_as_process"
      }
    },
    {
      "_embedded_ecs-established_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "established"
      }
    },
    {
      "_embedded_ecs-resumed_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "resumed"
      }
    },
    {
      "_embedded_ecs-max_bytes_per_process_exceeded_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "max_bytes_per_process_exceeded"
      }
    },
    {
      "_embedded_ecs-interactive_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "interactive"
      }
    },
    {
      "_embedded_ecs-exists_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "exists"
      }
    },
    {
      "_embedded_ecs-trusted_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "trusted"
      }
    },
    {
      "_embedded_ecs-valid_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "valid"
      }
    },
    {
      "_embedded_ecs-go_stripped_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "go_stripped"
      }
    },
    {
      "_embedded_ecs-coldstart_to_boolean": {
        "mapping": {
          "type": "boolean"
        },
        "match": "coldstart"
      }
    },
    {
      "_embedded_ecs-exports_to_flattened": {
        "mapping": {
          "type": "flattened"
        },
        "match": "exports"
      }
    },
    {
      "_embedded_ecs-structured_data_to_flattened": {
        "mapping": {
          "type": "flattened"
        },
        "match": "structured_data"
      }
    },
    {
      "_embedded_ecs-imports_to_flattened": {
        "mapping": {
          "type": "flattened"
        },
        "match": "*imports"
      }
    },
    {
      "_embedded_ecs-attachments_to_nested": {
        "mapping": {
          "type": "nested"
        },
        "match": "attachments"
      }
    },
    {
      "_embedded_ecs-segments_to_nested": {
        "mapping": {
          "type": "nested"
        },
        "match": "segments"
      }
    },
    {
      "_embedded_ecs-elf_sections_to_nested": {
        "path_match": "*.elf.sections",
        "mapping": {
          "type": "nested"
        }
      }
    },
    {
      "_embedded_ecs-pe_sections_to_nested": {
        "path_match": "*.pe.sections",
        "mapping": {
          "type": "nested"
        }
      }
    },
    {
      "_embedded_ecs-macho_sections_to_nested": {
        "path_match": "*.macho.sections",
        "mapping": {
          "type": "nested"
        }
      }
    }
  ],
  "properties": {
    "input": {
      "properties": {
        "type": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "@timestamp": {
      "ignore_malformed": false,
      "type": "date"
    },
    "ecs": {
      "properties": {
        "version": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "log": {
      "properties": {
        "file": {
          "properties": {
            "inode": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "path": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "device_id": {
              "ignore_above": 1024,
              "type": "keyword"
            },
            "fingerprint": {
              "index": false,
              "type": "keyword"
            }
          }
        },
        "offset": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "level": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "data_stream": {
      "properties": {
        "namespace": {
          "type": "constant_keyword"
        },
        "type": {
          "type": "constant_keyword"
        },
        "dataset": {
          "type": "constant_keyword"
        }
      }
    },
    "event": {
      "properties": {
        "original": {
          "index": false,
          "type": "keyword",
          "doc_values": false
        },
        "module": {
          "type": "constant_keyword",
          "value": "filestream"
        },
        "dataset": {
          "type": "constant_keyword"
        }
      }
    },
    "message": {
      "type": "match_only_text"
    },
    "tags": {
      "ignore_above": 1024,
      "type": "keyword"
    }
  }
}

Related issues

## Screenshots

@belimawr belimawr self-assigned this Mar 6, 2026
@belimawr belimawr added bug Something isn't working, use only for issues Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane] labels Mar 6, 2026
@belimawr belimawr marked this pull request as ready for review March 6, 2026 21:52
@belimawr belimawr requested a review from a team as a code owner March 6, 2026 21:52
@elasticmachine
Copy link

Pinging @elastic/elastic-agent-data-plane (Team:Elastic-Agent-Data-Plane)

@elasticmachine
Copy link

💚 Build Succeeded

cc @belimawr

@belimawr belimawr merged commit 58e608f into elastic:main Mar 12, 2026
9 checks passed
@elastic-vault-github-plugin-prod

Package filestream - 2.3.3 containing this change is available at https://epr.elastic.co/package/filestream/2.3.3/

@andrewkroh andrewkroh added the Integration:filestream Custom Logs (Filestream) label Mar 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working, use only for issues Integration:filestream Custom Logs (Filestream) Team:Elastic-Agent-Data-Plane Agent Data Plane team [elastic/elastic-agent-data-plane]

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Filestream Input: creating all duplicated embedded ECS dynamic templates

5 participants