Skip to content

Commit

Permalink
Merge remote-tracking branch 'upstream/master' into feature/use-debian-9
Browse files Browse the repository at this point in the history 
* upstream/master:
  packer cache support for the 7.x and 7.latestMinor branches (elastic#25091)
  Remove EventFetcher and EventsFetcher interface (elastic#25093)
  Update go-structform to 0.0.8 (elastic#25051)
  Update copy_fields.asciidoc (elastic#25053)
  [elastic-agent] ensure container is backwards compatible (elastic#25092)
  Add --fleet-server-service-token. Rename --fleet-server to --fleet-server-es. (elastic#25083)
  Add cgroup.cpuacct percentages (elastic#25057)
  Add tests for truncated and symlinked files in filestream input (elastic#24425)
  Fix panic when Hearbeat monitor initialization fails twice (elastic#25073)
  [Filebeat][httpjson] Change append transform to initiate new fields as a slice (elastic#25074)
  Osquerybeat: Result values type translation (elastic#25012)
  Update Osquerybeat spec to get it downloading from the correct artifactory path (elastic#25076)
  Fix changelog (elastic#25079)
  Strip Azure EventHub connection string in debug logs (elastic#25066)
  Change googlecloud to gcp in field names (elastic#25038)
  Bump stack version to 7.12.0 for testing (elastic#24957)
  packer-cache: cache the existing docker images on ARM and some more (elastic#25068)
  Disable logstash TestFetch flaky test (elastic#25044)
  • Loading branch information
@v1v
v1v committed Apr 15, 2021
2 parents f71dddc + ae02c5f commit 28b46f9
Show file tree
Hide file tree
Showing 62 changed files with 4,135 additions and 3,605 deletions.
131 changes: 65 additions & 66 deletions .ci/packer_cache.sh
100644 → 100755
View file
Original file line number Diff line number Diff line change
Expand Up @@ -9,79 +9,78 @@ source /usr/local/bin/bash_standard_lib.sh
# shellcheck disable=SC1091
source ./dev-tools/common.bash

# Docker images used on Dockerfiles 2019-07-12
# aerospike:3.9.0
# alpine:edge
# apache/couchdb:1.7
# busybox:latest
# ceph/daemon:master-6373c6a-jewel-centos-7-x86_64
# cockroachdb/cockroach:v19.1.1
# consul:1.4.2
# coredns/coredns:1.5.0
# couchbase:4.5.1
# debian:latest
# debian:stretch
# docker.elastic.co/beats-dev/fpm:1.11.0
# docker.elastic.co/beats/metricbeat:6.5.4
# docker.elastic.co/beats/metricbeat:7.2.0
# docker.elastic.co/elasticsearch/elasticsearch:7.2.0
# docker.elastic.co/kibana/kibana:7.2.0
# docker.elastic.co/logstash/logstash:7.2.0
# docker.elastic.co/observability-ci/database-instantclient:12.2.0.1
# envoyproxy/envoy:v1.7.0
# exekias/localkube-image
# haproxy:1.8
# httpd:2.4.20
# java:8-jdk-alpine
# jplock/zookeeper:3.4.8
# maven:3.3-jdk-8
# memcached:1.4.35-alpine
# microsoft/mssql-server-linux:2017-GA
# mongo:3.4
# mysql:5.7.12
# nats:1.3.0
# nginx:1.9
# oraclelinux:7
# postgres:9.5.3
# prom/prometheus:v2.6.0
# python:3.6-alpine
# quay.io/coreos/etcd:v3.3.10
# rabbitmq:3.7.4-management
# redis:3.2.12-alpine
# redis:3.2.4-alpine
# store/oracle/database-enterprise:12.2.0.1
# traefik:1.6-alpine
# tsouza/nginx-php-fpm:php-7.1
# ubuntu:16.04
# ubuntu:trusty
######################
############ FUNCTIONS
######################
function getBeatsVersion() {
grep 'defaultBeatVersion' libbeat/version/version.go | cut -d= -f2 | sed 's#"##g' | tr -d " "
}

get_go_version

DOCKER_IMAGES="docker.elastic.co/observability-ci/database-instantclient:12.2.0.1
docker.elastic.co/observability-ci/database-enterprise:12.2.0.1
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-arm
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-darwin
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main-debian7
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main-debian8
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main-debian9
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-mips
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-ppc
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-s390x
golang:${GO_VERSION}
"
if [ -x "$(command -v docker)" ]; then
for image in ${DOCKER_IMAGES}
do
(retry 2 docker pull ${image}) || echo "Error pulling ${image} Docker image, we continue"
function dockerPullCommonImages() {
DOCKER_IMAGES="docker.elastic.co/observability-ci/database-instantclient:12.2.0.1
docker.elastic.co/observability-ci/database-enterprise:12.2.0.1
docker.elastic.co/beats-dev/fpm:1.11.0
golang:1.14.12-stretch
centos:7
"
for image in ${DOCKER_IMAGES} ; do
(retry 2 docker pull ${image}) || echo "Error pulling ${image} Docker image. Continuing."
done

docker tag \
docker.elastic.co/observability-ci/database-instantclient:12.2.0.1 \
store/oracle/database-instantclient:12.2.0.1 \
|| echo "Error setting the Oracle Instant Client tag"
docker tag \
docker.elastic.co/observability-ci/database-enterprise:12.2.0.1 \
store/oracle/database-enterprise:12.2.0.1 \
|| echo "Error setting the Oracle Dtabase tag"
|| echo "Error setting the Oracle Database tag"
}

function dockerPullImages() {
SNAPSHOT=$1
get_go_version

DOCKER_IMAGES="
docker.elastic.co/elasticsearch/elasticsearch:${SNAPSHOT}
docker.elastic.co/kibana/kibana:${SNAPSHOT}
docker.elastic.co/logstash/logstash:${SNAPSHOT}
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-arm
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-base-arm-debian9
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-darwin
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main-debian7
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main-debian8
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-main-debian9
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-mips
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-ppc
docker.elastic.co/beats-dev/golang-crossbuild:${GO_VERSION}-s390x
golang:${GO_VERSION}
"
for image in ${DOCKER_IMAGES}
do
(retry 2 docker pull ${image}) || echo "Error pulling ${image} Docker image. Continuing."
done
}

#################
############ MAIN
#################
if [ -x "$(command -v docker)" ]; then
set -x
echo "Docker pull common docker images"
dockerPullCommonImages

## GitHub api returns up to 100 entries.
## Probably we need a different approach to search the latest minor.
latestMinor=$(curl -s https://api.github.com/repos/elastic/beats/branches\?per_page=100 | jq -r '.[].name' | grep "^7." | tail -1)

for branch in master 7.x $latestMinor ; do
if [ "$branch" != "master" ] ; then
echo "Checkout the branch $branch"
git checkout "$branch"
fi

VERSION=$(getBeatsVersion)
dockerPullImages "${VERSION}-SNAPSHOT"
done
fi
1 change: 1 addition & 0 deletions CHANGELOG-developer.next.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ The list below covers the major changes between 7.0.0-rc2 and master only.
- Remove global ACK handler support via `SetACKHandler` from publisher pipeline. {pull}19632[19632]
- Make implementing `Close` required for `reader.Reader` interfaces. {pull}20455[20455]
- Remove `NumCPU` as clients should update the CPU count on the fly in case of config changes in a VM. {pull}23154[23154]
- Remove Metricbeat EventFetcher and EventsFetcher interface. Use the reporter interface instead. {pull}25093[25093]

==== Bugfixes

Expand Down
5 changes: 5 additions & 0 deletions CHANGELOG.next.asciidoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Rename `network.direction` values in crowdstrike/falcon to `ingress`/`egress`. {pull}23041[23041]
- Possible values for Netflow's locality fields (source.locality, destination.locality and flow.locality) are now `internal` and `external`, instead of `private` and `public`. {issue}24272[24272] {pull}24295[24295]
- Add User Agent Parser for Azure Sign In Logs Ingest Pipeline {pull}23201[23201]
- Changes filebeat httpjson input's append transform to create a list even with only a single value{pull}25074[25074]

*Heartbeat*

Expand Down Expand Up @@ -160,6 +161,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d

*Affecting all Beats*

- Fix events being dropped if they contain a floating point value of NaN or Inf. {pull}25051[25051]
- Fix templates being overwritten if there was an error when check for the template existance. {pull}24332[24332]
- Fix Kubernetes autodiscovery provider to correctly handle pod states and avoid missing event data {pull}17223[17223]
- Fix `add_cloud_metadata` to better support modifying sub-fields with other processors. {pull}13808[13808]
Expand Down Expand Up @@ -279,6 +281,7 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix a connection error in httpjson input. {pull}16123[16123]
- Fix integer overflow in S3 offsets when collecting very large files. {pull}22523[22523]
- Fix CredentialsJSON unpacking for `gcp-pubsub` and `httpjson` inputs. {pull}23277[23277]
- Strip Azure Eventhub connection string in debug logs. {pulll}25066[25066]

*Filebeat*

Expand Down Expand Up @@ -392,11 +395,13 @@ https://github.com/elastic/beats/compare/v7.0.0-alpha2...master[Check the HEAD d
- Fix S3 input validation for non amazonaws.com domains. {issue}24420[24420] {pull}24861[24861]
- Fix google_workspace and okta modules pagination when next page template is empty. {pull}24967[24967]
- Fix IPtables Pipeline and Ubiquiti dashboard. {issue}24878[24878] {pull}24928[24928]
- Fix gcp module field names to use gcp instead of googlecloud. {pull}25038[25038]

*Heartbeat*

- Fixed excessive memory usage introduced in 7.5 due to over-allocating memory for HTTP checks. {pull}15639[15639]
- Fixed TCP TLS checks to properly validate hostnames, this broke in 7.x and only worked for IP SANs. {pull}17549[17549]
- Fix panic when initialization of ICMP monitors fail twice. {pull}25073[25073]

*Journalbeat*

Expand Down
Loading

0 comments on commit 28b46f9

Please sign in to comment.