Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/master' into feature/e2e-fleet
Browse files Browse the repository at this point in the history 
* upstream/master: (73 commits)
  Remove GCP support from Functionbeat (elastic#28253)
  Move labels and annotations under kubernetes.namespace. (elastic#27917)
  Update go release version 1.17.1 (elastic#27543)
  Osquerybeat: Runner and Fetcher unit tests (elastic#28290)
  Osquerybeat: Improve handling of osquery.autoload file, allow customizations (elastic#28289)
  seccomp: allow clone3 syscall for x86 (elastic#28117)
  packetbeat/protos/dns: don't render missing A and AAAA addresses from truncated records (elastic#28297)
  [7.x] [DOCS] Update api_key example on elasticsearch output (elastic#28288)
  [cloud][docker] use the private docker namespace (elastic#28286)
  Update aws-lambda-go library version to 1.13.3 (elastic#28236)
  Deprecate common.Float (elastic#28280)
  Filebeat: Change compatibility test stage to test against previous minor instead of 7.11 (elastic#28274)
  x-pack/filebeat/module/threatintel/misp: add support for secondary object attribute handling (elastic#28124)
  Explicitly pass http config to doppler consumer (elastic#28277)
  processors/actions/add_fields: Do not panic if event.Fields is nil map (elastic#28219)
  Resolved timestamp for defender atp (elastic#28272)
  [Winlogbeat] Tolerate faults when Windows Event Log session is interrupted (elastic#28191)
  [elastic-agent] proxy requests to subprocesses to their metrics endpoints (elastic#28165)
  Build cloud docker images for elastic-agent (elastic#28134)
  Upgrade k8s go-client library (elastic#28228)
  ...
  • Loading branch information
@v1v
v1v committed Oct 11, 2021
2 parents 10d1617 + 3754124 commit 86c1cf8
Show file tree
Hide file tree
Showing 1,596 changed files with 46,341 additions and 160,399 deletions.
2 changes: 1 addition & 1 deletion .backportrc.json
View file
@@ -1,6 +1,6 @@
{
"upstream": "elastic/beats",
"branches": [ { "name": "7.x", "checked": true }, "7.15", "7.14", "7.13", "7.12" ],
"branches": [ { "name": "7.x", "checked": true }, "7.15", "7.14" ],
"labels": ["backport"],
"autoAssign": true,
"prTitle": "Cherry-pick to {targetBranch}: {commitMessages}"
Expand Down
28 changes: 10 additions & 18 deletions .ci/packaging.groovy
View file
Expand Up @@ -40,8 +40,6 @@ pipeline {
}
parameters {
booleanParam(name: 'macos', defaultValue: false, description: 'Allow macOS stages.')
booleanParam(name: 'linux', defaultValue: true, description: 'Allow linux stages.')
booleanParam(name: 'arm', defaultValue: true, description: 'Allow ARM stages.')
}
stages {
stage('Filter build') {
Expand Down Expand Up @@ -125,12 +123,6 @@ pipeline {
stage('Package Linux'){
agent { label 'ubuntu-18 && immutable' }
options { skipDefaultCheckout() }
when {
beforeAgent true
expression {
return params.linux
}
}
environment {
HOME = "${env.WORKSPACE}"
PLATFORMS = [
Expand Down Expand Up @@ -224,12 +216,6 @@ pipeline {
stage('Package Docker images for linux/arm64'){
agent { label 'arm' }
options { skipDefaultCheckout() }
when {
beforeAgent true
expression {
return params.arm
}
}
environment {
HOME = "${env.WORKSPACE}"
PACKAGES = "docker"
Expand Down Expand Up @@ -335,26 +321,31 @@ def tagAndPush(Map args = [:]) {
}
// supported image flavours
def variants = ["", "-oss", "-ubi8"]
//

if(beatName == 'elastic-agent'){
variants.add("-complete")
variants.add("-cloud")
}

variants.each { variant ->
// cloud docker images are stored in the private docker namespace.
def sourceNamespace = variant.equals('-cloud') ? 'beats-ci' : 'beats'
tags.each { tag ->
// TODO:
// For backward compatibility let's ensure we tag only for amd64, then E2E can benefit from until
// they support the versioning with the architecture
if ("${arch}" == "amd64") {
doTagAndPush(beatName: beatName, variant: variant, sourceTag: libbetaVer, targetTag: "${tag}")
doTagAndPush(beatName: beatName, variant: variant, sourceTag: libbetaVer, targetTag: "${tag}", sourceNamespace: sourceNamespace)
}
doTagAndPush(beatName: beatName, variant: variant, sourceTag: libbetaVer, targetTag: "${tag}-${arch}")
doTagAndPush(beatName: beatName, variant: variant, sourceTag: libbetaVer, targetTag: "${tag}-${arch}", sourceNamespace: sourceNamespace)
}
}
}

/**
* @param beatName name of the Beat
* @param variant name of the variant used to build the docker image name
* @param sourceNamespace namespace to be used as source for the docker tag command
* @param sourceTag tag to be used as source for the docker tag command, usually under the 'beats' namespace
* @param targetTag tag to be used as target for the docker tag command, usually under the 'observability-ci' namespace
*/
Expand All @@ -363,7 +354,8 @@ def doTagAndPush(Map args = [:]) {
def variant = args.variant
def sourceTag = args.sourceTag
def targetTag = args.targetTag
def sourceName = "${DOCKER_REGISTRY}/beats/${beatName}${variant}:${sourceTag}"
def sourceNamespace = args.sourceNamespace
def sourceName = "${DOCKER_REGISTRY}/${sourceNamespace}/${beatName}${variant}:${sourceTag}"
def targetName = "${DOCKER_REGISTRY}/observability-ci/${beatName}${variant}:${targetTag}"
def iterations = 0
retryWithSleep(retries: 3, seconds: 5, backoff: true) {
Expand Down
2 changes: 1 addition & 1 deletion .go-version
View file
@@ -1 +1 @@
1.16.6
1.17.1
101 changes: 49 additions & 52 deletions .mergify.yml
View file
Expand Up @@ -12,45 +12,6 @@ pull_request_rules:
labels:
- "backport"
title: "[{{ destination_branch }}](backport #{{ number }}) {{ title }}"
- name: backport patches to 7.x branch
conditions:
- merged
- label=backport-v7.16.0
actions:
backport:
assignees:
- "{{ author }}"
branches:
- "7.x"
labels:
- "backport"
title: "[{{ destination_branch }}](backport #{{ number }}) {{ title }}"
- name: backport patches to 7.13 branch
conditions:
- merged
- label=backport-v7.13.0
actions:
backport:
assignees:
- "{{ author }}"
branches:
- "7.13"
labels:
- "backport"
title: "[{{ destination_branch }}](backport #{{ number }}) {{ title }}"
- name: backport patches to 7.12 branch
conditions:
- merged
- label=backport-v7.12.0
actions:
backport:
assignees:
- "{{ author }}"
branches:
- "7.12"
labels:
- "backport"
title: "[{{ destination_branch }}](backport #{{ number }}) {{ title }}"
- name: ask to resolve conflict
conditions:
- conflict
Expand Down Expand Up @@ -106,19 +67,6 @@ pull_request_rules:
- files~=^\.go-version$
actions:
delete_head_branch:
- name: backport patches to 7.14 branch
conditions:
- merged
- label=backport-v7.14.0
actions:
backport:
assignees:
- "{{ author }}"
branches:
- "7.14"
labels:
- "backport"
title: "[{{ destination_branch }}](backport #{{ number }}) {{ title }}"
- name: automatic approval for mergify pull requests with changes in bump-rules
conditions:
- author=mergify[bot]
Expand Down Expand Up @@ -151,6 +99,55 @@ pull_request_rules:
- files~=^\.mergify\.yml$
actions:
delete_head_branch:
- name: notify the backport policy
conditions:
- -label~=^backport
- base=master
actions:
comment:
message: |
This pull request does not have a backport label. Could you fix it @{{author}}? 🙏
To fixup this pull request, you need to add the backport labels for the needed
branches, such as:
* `backport-v./d./d./d` is the label to automatically backport to the `7./d` branch. `/d` is the digit
**NOTE**: `backport-skip` has been added to this pull request.
label:
add:
- backport-skip
- name: remove-backport label
conditions:
- label~=backport-v
actions:
label:
remove:
- backport-skip
- name: backport patches to 7.x branch
conditions:
- merged
- label=backport-v7.16.0
actions:
backport:
assignees:
- "{{ author }}"
branches:
- "7.x"
labels:
- "backport"
title: "[{{ destination_branch }}](backport #{{ number }}) {{ title }}"
- name: backport patches to 7.14 branch
conditions:
- merged
- label=backport-v7.14.0
actions:
backport:
assignees:
- "{{ author }}"
branches:
- "7.14"
labels:
- "backport"
title: "[{{ destination_branch }}](backport #{{ number }}) {{ title }}"
- name: backport patches to 7.15 branch
conditions:
- merged
Expand Down
7 changes: 7 additions & 0 deletions CHANGELOG-developer.next.asciidoc
View file
Expand Up @@ -63,6 +63,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
- Fix export dashboard command when running against Elastic Cloud hosted Kibana. {pull}22746[22746]
- Remove `event.dataset` (ECS) annotion from `libbeat.logp`. {issue}27404[27404]
- Errors should be thrown as errors. Metricsets inside Metricbeat will now throw errors as the `error` log level. {pull}27804[27804]
- Avoid panicking in `add_fields` processor when input event.Fields is a nil map. {pull}28219[28219]

==== Added

Expand Down Expand Up @@ -120,3 +121,9 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
- Add sorting to array fields for generated data files (*-generated.json) {pull}25320[25320]
- Update to go-concert 0.2.0 {pull}27162[27162]
- Update Go version to 1.16.5. {issue}26182[26182] {pull}26186[26186]
- Introduce `libbeat/beat.Beat.OutputConfigReloader` {pull}28048[28048]
- Update Go version to 1.17.1. {pull}27543[27543]

==== Deprecated

- Deprecated the `common.Float` type. {issue}28279[28279] {pull}28280[28280]
93 changes: 93 additions & 0 deletions CHANGELOG.asciidoc
View file
Expand Up @@ -12,6 +12,99 @@ Changes will be described in a later alpha / beta.
=== Beats version 8.0.0-alpha1

Changes will be described in a later alpha / beta.
[[release-notes-7.15.0]]
=== Beats version 7.15.0
https://github.com/elastic/beats/compare/v7.14.2...v7.15.0[View commits]

==== Breaking changes

*Affecting all Beats*

- Loading Kibana assets (dashboards, index templates) rely on Saved Object API. So to provide a reliable service, Beats can only import and export dashboards using at least Kibana 7.15. {issue}20672[20672] {pull}27220[27220]

*Filebeat*

- Remove all alias fields pointing to ECS fields from modules. This affects the Suricata and Traefik modules. {issue}10535[10535] {pull}26627[26627]
- Fix Crowdstrike ingest pipeline that was creating flattened `process` fields. {issue}27622[27622] {pull}27623[27623]
- Rename `log.path` to `log.file.path` in filestream to be consistent with `log` input and ECS. {pull}27761[27761]

*Heartbeat*
- Remove long deprecated `watch_poll` functionality. {pull}27166[27166]
- Fix inconsistency in `event.dataset` values between heartbeat and fleet by always setting this value to the monitor type / fleet dataset. {pull}27535[27535]

*Metricbeat*

- Fix Elasticsearch jvm.gc.collectors.old being exposed as young {issue}19636[19636] {pull}26616[26616]

==== Bugfixes

*Affecting all Beats*

- Improve `perfmon` metricset performance. {pull}26886[26886]
- Preserve annotations in a kubernetes namespace metadata {pull}27045[27045]
- Fix build constraint that caused issues with doc builds. {pull}27381[27381]
- Do not try to load ILM policy if `check_exists` is `false`. {pull}27508[27508] {issue}26322[26322]
- Fix bug with cgroups hierarchy override path in cgroups {pull}27620[27620]
- Beat `setup kibana` command may use the elasticsearch API key defined in `output.elasticsearch.api_key`. {issue}24015[24015] {pull}27540[27540]
- Fix `decode_xml` handling of array merging when using `to_lower: true`. {pull}27922[27922]
- Separate namespaces for V1 and V2 controller paths {pull}27676[27676]
- Do not try to load ILM policy if `check_exists` is `false`. {pull}27508[27508] {issue}26322[26322]
- Kubernetes autodiscover fails in node scope if node name cannot be discovered {pull}26947[26947]

*Auditbeat*

- File Integrity Module: Honor `include_files` when doing initial scan. {issue}27273[27273] {pull}27722[27722]

*Filebeat*

- Update Filebeat compatibility function to remove processor description field on ES < 7.9.0 {pull}27774[27774]
- Make filestream events ECS compliant. {issue}27776[27776]

*Metricbeat*

- Allow metric prefix override per service in gcp module. {pull}26960[26960]
- Update metrics configuration and dashboards after changes in the Azure Monitor {pull}27520[27520]

*Winlogbeat*

- Fix an issue with message template caching in the `wineventlog-experimental` API implementation. {pull}26826[26826]

==== Added

*Affecting all Beats*

- Add proxy support for AWS functions. {pull}26832[26832]
- Added policies to the Elasticsearch output for non indexible events {pull}26952[26952]
- Add `logging.metrics.namespaces` config option to control what metric groups are reported in logs. {pull}25727[25727]
- Add sha256 digests to RPM packages. {issue}23670[23670]
- Add new 'offline' docker image for Elastic Agent. {pull}27052[27052]
- Add cgroups V2 support {pull}27242[27242]
- Update ECS field definitions to ECS 1.11.0. {pull}27107[27107]
- The disk queue is now GA. {pull}27515[27515]
- Add `daemonset.name` in pods controlled by DaemonSets {pull}26808[26808], {issue}25816[25816]

*Filebeat*

- Add new template functions and `value_type` parameter to `httpjson` transforms. {pull}26847[26847]
- Add support to merge registry updates in the filestream input across multiple ACKed batches in case of backpressure in the registry or disk. {pull}25976[25976]
- Add support to `decode_cef` for MAC addresses that do not contain separator characters. {issue}27050[27050] {pull}27109[27109]
- Add new `hmac` template function for httpjson input {pull}27168[27168]
- Update `tags` and `threatintel.indicator.provider` fields in `threatintel.anomali` ingest pipeline {issue}24746[24746] {pull}27141[27141]
- Move AWS module and filesets to GA. {pull}27428[27428]
- Update ecs.version to ECS 1.11.0. {pull}27107[27107]
- Add option for S3 input to work without SQS notification {issue}18205[18205] {pull}27332[27332]

*Metricbeat*

- Move openmetrics module to oss. {pull}26561[26561]
- Fix release state of kubernetes metricsets. {pull}26864[26864]
- Add `gke` metricset collection to `gcp` module {pull}26824[26824]
- Added `statsd.mappings` configuration for Statsd module {pull}26220[26220]
- Added Airflow lightweight module {pull}26220[26220]
- Add state_job metricset to Kubernetes module{pull}26479[26479]
- Bump AWS SDK version to v0.24.0 for WebIdentity authentication flow {issue}19393[19393] {pull}27126[27126]


[[release-notes-7.14.2]]
=== Beats version 7.14.2
https://github.com/elastic/beats/compare/v7.14.1...v7.14.2[View commits]
Expand Down

0 comments on commit 86c1cf8

Please sign in to comment.