All information related to ECS is versioned in the elastic/ecs repository. All changes to ECS happen through Pull Requests submitted through Git.
ECS is an open source project and we love to receive contributions from our community - you!
- How to contribute
- Git and Github Guidelines
- Feature freezes and branching
- Documentation
- Schema Files
- Additional Resources
In April 2023, OpenTelemetry and Elastic made an important joint announcement. In this announcement Elastic shared its intention to achieve convergence of ECS and OTel Semantic Conventions into a single standard maintained by OpenTelemetry.
The stated plan has been to keep ECS in a frozen state during the transition. However, it is also apparent that these things take time. It takes time for the OTel community to adopt donated fields, and it will take time for development teams to build OTel native constructs in the Elastic stack. In the meantime, ECS users need to be able to develop features for Elastic that rely on continued contributions to the schemas that drive our technology.
For these reasons, we need a process and guidelines for contributing to these data schemas during this period that allows us to avoid breaking changes.
Historically, ECS has shipped a new version with every minor release of the Elastic stack. While the schema is effectively frozen during the Otel donation period, this approach has been halted.
Moving forward, we will release ECS at the team's discretion as new material changes to the schema are adopted.
While the decision to release will be discretionary, any release will still coincide with a minor Elastic stack release; however, not every minor version will warrant a new release of the stack.
Bug fixes or minor field addition changes can be made directly to the ECS project and submitted as pull requests.
Significant changes that add new use cases, top-level fieldsets, or could be considered controversial are considered material. The general rule for contributing new material changes to schemas during the transition period is
- First, merge a pull request to OTel Semantic Conventions with new fields, namespaces or schemas
- Second, to backport those changes to ECS at the starting point indicated in the table below
- Finally, once the Semantic Conventions changes are marked as stable, remove the Beta designation in ECS
This will ensure that the latest changes are included in OTel Semantic Conventions, where schema evolution will continue as the merger proceeds. It will also allow teams and users to continue using ECS while OTel migration tools and guidance are being developed. Finally, this will reduce the risk of breaking changes if new fields are merged first to ECS, and then require changes before being adopted in Semantic Conventions.
There are some exceptions to this rule.
- My contribution to OTel Semantic Conventions is stalled. We are waiting for a sign-off from a second company. In the meantime, our Elastic feature is blocked.
- I want to build a workflow in Elastic, and the fields I need to proceed are already in OTel but not in ECS where I need them today.
In these cases, the recommendation is to make a contribution to ECS to unblock development. The appropriate ECS starting point can be an RFC or pull request based on the maturity of the Otel changes. Please see the following table.
OTel submission maturity | Breaking changes expected | ECS starting point |
---|---|---|
OTel working groups accepts the premise of the addition and commits to considering this proposal as it advances. | Major | RFC Stage 1 |
The initial field definitions comprehensively model the addition to the schema. Fundamental questions and concerns are resolved, though some less significant questions remain open. | Iterative | RFC Stage 2 |
All requested changes from codeowners have been addressed, and there are no open discussions. | Iterative | Open an ECS pull request with new fields marked Beta |
Fields, schema, namespace exists in OTel and are designated experimental | Iterative | Open an ECS pull request with new fields marked Beta |
Fields, schema, namespace exists in OTel and are designated stable | None | Open an ECS pull request with new fields marked GA |
You need these tools to contribute to the ECS repo:
- Sign the Contributor License Agreement.
- Set up your git environment.
- Create your own fork of the ECS repo.
- Clone your fork to your machine.
- Create a local branch to hold your changes.
- Run
git checkout -b branch-name
, wherebranch-name
is the name you want to give your local branch
- Run
- Do your work.
- Schema changes will be done in the
.yml
files under theschemas
directory. Review the Schema Files section below. - Generator scripts and other tooling reside in the
scripts
directory. - Generated artifacts fall under the
generated
directory. - Documentation files are located in the
docs
directory.
- Schema changes will be done in the
- Run
make
to update generated files. - If necessary, make sure tests pass.
- Run
make test
- Add tests for your changes, if necessary
- Run
- Commit your changes locally.
- Run
git commit -a -m "your message"
- Run
- Push your changes to your own github.com fork.
- Run
git push --set-upstream origin branch-name
- In this command,
origin
is an alias that references your fork.
- Run
- Request feedback about your changes.
- Create a Pull Request against the ECS repo.
- (Look for the
Compare & pull request
button on your branch in github.com.)
- (Look for the
- Add an entry to CHANGELOG.next.md.
- Wait for reviews on your PR.
- Incorporate review comments and push updates if needed.
- Create a Pull Request against the ECS repo.
- Thank you for your contribution!
Important: Be sure to push changes only to your own fork. Changes must be approved before they are merged into the main repository.
We follow the Github forking model for collaboration in the ECS repo. Typically contributors will add a remote repository called upstream
to point to the Elastic ECS repo to add latest changes from the repo to their fork.
- When submitting a PR for review, please perform and interactive rebase to clean up the commit history. A logical commit history is easier for reviewers to follow.
- Use meaningful and helpful commit messages on your changes and an explanation of why you made those changes.
- When merging, maintainers will squash your commits into a single commit.
Please follow these guidelines when submitting PRs:
- Include an explanation of your changes in the PR description.
- Links to relevant issues, external resources, or related PRs are helpful and useful.
- Update any tests or add new tests where appropriate.
Please follow these guidelines when submitting Issues:
- Go to the ECS repo: https://github.com/elastic/ecs
- Click
Issues
in the nav bar under the repo name. - Click
New issue
. Provide as many details as possible to help reviewers and other contributors understand your proposal. - Add your text, and click
Submit new issue
.
For an upcoming release, ECS uses a feature freeze (FF) approach. A release branch is cut from main
for an upcoming
release. When a branch is frozen, changes are limited to bug fixes or doc updates.
Any schema changes or tooling updates will be merged into main
for the next ECS version.
ECS maintains two changelog files:
- CHANGELOG.md contains a list of notable changes for each released version of ECS.
- CHANGELOG.next.md contains a list of unreleased ECS changes.
Breaking changes intended for the next major version should be included underneath the Breaking changes
sections in CHANGELOG.next.md
.
ECS maintains multiple release branches in the repo. The main
branch is where all new contributions should be submitted, and features and bug fixes will be backported into other branches when appropriate. Any backporting needs will be handled by the ECS team.
Refer to the backport tool's repo for requirements and install guide. A project config is maintained in the root of this repo.
Run:
$ npx backport --pr 1234
Select target branch(es) to backport to:
? Select commit #1234 (cb79e8f5)
? Select branch (Press <space> to select, <a> to toggle all, <i> to invert selection)
❯◯ 8.1
◯ 8.0
◯ 1.12
...
New PR(s) will be opened against the targeted branch(es).
ECS documentation is written in asciidoc format in the docs/
directory.
To build the docs and open them in your browser:
make docs
The following files are generated based on the current schema using Jinja templates:
File | Template |
---|---|
fields.asciidoc | fields_template.j2 |
fields-values.asciidoc | field_values_template.j2 |
field-details.asciidoc | field_details.j2 |
Running make
will update these files using the scripts/generators/asciidoc_fields.py generator. These doc files should not be modified directly. Any changes as a result of a schema update and subsequent run of make
should be committed.
Jinja templates allow for formatting or styling changes to templates without needing to modify the generator script directly. Some details about the Jinja template engine and our implementation are covered below as a primer; the full syntax and semantics of Jinja is covered in the project's documentation.
- Statements:
{% ... %}
- Expressions:
{{ ... }}
- Comments:
{{# ... #}}
Whitespace can be stripped by adding the minus sign (-
) to the start or end of a block. Adding -
to the start or end of a block will remove the whitespace before or after that block.
{% for i in numbers -%}
{{i}}
{%- endfor %}
All elements would be rendered without any separating whitespace. If numbers
is list of numbers from 0
to 9
, the output would be 0123456789
.
Templates variables are passed to the template by the application. Typically these will either be used in an expression or within a control structure statement (e.g. a for
loop). In the below example, users
is passed into the template and is iterated over with a for
loop.
<ul>
{% for user in users %}
<li>{{ user }}</li>
{% endfor %}
</ul>
The @templated('template_file_name')
decorator is used to inject the additional functionality that renders and returns the template's content to the generator. Decorated functions should return a dict used to generate the template. When the decorated function returns, the dictionary is passed to the template renderer.
@templated('fields_template.j2')
def page_field_index(intermediate_nested, ecs_version):
fieldsets = ecs_helpers.dict_sorted_by_keys(intermediate_nested, ['group', 'name'])
return dict(ecs_version=ecs_version, fieldsets=fieldsets)
The schemas directory contains the files which define the Elastic Common Schema data model. The file structure is documented in schemas/README.md. Field additions and modifications will be made to the schemas/*.yml
files.
Users consuming ECS to generate something for other use cases should use the generated/ecs/*.yml
files. More detail can be found here.
The schemas/subsets directory contains the configuration to control advanced field nesting use cases.
The config is used with the --subset
option to control which field sets or specific fields appear in the final generated
artifacts.