Migrate obs-infraobs-integrations to package-spec v3 #2

[shmsr]

Migrate a batch of packages from v1/v2 to v3 package-spec. Used the following scripts and some manual changes to generate the changes in this PR:

yq_hack.zsh

#!/bin/bash

set -e
set -u
set -o pipefail
set -x

pushd $1

find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].release" {} \;
find data_stream -type f -path "*fields*" -name "ecs.yml" -exec yq e -i "unique_by(.name)" {} \;
find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].fields.[].required" {} \;

yq e -i 'del .release' manifest.yml
yq e -i "with(select(.license != null); .conditions.elastic.subscription = .license) | del .license" manifest.yml

popd

git add -u $1/
git commit -m "[$1]: migration with yq"

ecs-update.zsh

#!/bin/bash

set -u
set -o pipefail
set -x

echo "-- executing ecs-update for $1 --"

ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1

if (( $? )) 
then
    echo "-- write validation for $1 --"
    pushd $1

    go run ../scripting/write_validation.go

    popd

    # Try again?
    git checkout .
    ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1
fi

And another Go program to write the validation.yml by parsing the elastic-package lint errors.

Then I hooked this scripts to run with:

$ pkgs=($(cat .github/CODEOWNERS | grep '@elastic/obs-infraobs-integrations' | cut -d ' ' -f 1 | cut -d '/' -f 3 | sort | uniq))

$ for ((idx=16; idx<=30; idx++))
do
scripting/yq_hack.zsh $pkgs[idx]
done

$ for ((idx=16; idx<=30; idx++))
do
scripting/ecs_update.zsh $pkgs[idx] || break
done

Another variation of the above script:

$ for ((idx=16; idx<=30; idx++))
do
case $pkgs[idx] in
  "nagios_xi" | "ibmmq" | "haproxy") echo "-- skip $pkgs[idx] --";;
  "nginx" | "memcached" | "kafka_log" | "jolokia_input" | "mongodb" | "mysql" | "microsoft_sqlserver" | "kafka" | "influxdb" | "iis" | "golang" | "hadoop") echo "-- done $pkgs[idx] --";;
  *) scripting/ecs_update.zsh $pkgs[idx] || break;;
esac
done

As I am creating a batch of 15 packages per migration, I am using a for loop. In case the migration is not possible for some packages, I put it in the first case to skip them.

Proposed commit message

Migrate packages owned by obs-infraobs-integrations but few couldn't be migrated. Marking them as follows:

• golang
• hadoop
• haproxy
• ibmmq
• iis
• influxdb
• jolokia_input
• kafka
• kafka_log
• memcached
• microsoft_sqlserver
• mongodb
• mysql
• nagios_xi
• nginx

Please see the comment #8171 (comment) in this PR itself to understand why some packages couldn't be migrated and also learn about the challenges faced when migrating.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Relates to #8170

Screenshots

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Start Time: 2023-10-19T07:50:38.693+0000

• Duration: 24 min 30 sec

Test stats 🧪

Test	Results
Failed	0
Passed	223
Skipped	0
Total	223

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

• /test : Re-trigger the build.

[elasticmachine]

🌐 Coverage report

Name	Metrics % (covered/total)	Diff
Packages	100.0% (17/17)	💚
Files	91.304% (21/23)	👎 -8.696
Classes	91.304% (21/23)	👎 -8.696
Methods	88.066% (214/243)	👍 12.308
Lines	94.99% (2977/3134)	👎 -5.01
Conditionals	100.0% (0/0)	💚

[shmsr]

• haproxy

  • Legacy viz: file "/Users/subhamsarkar/go/src/github.com/elastic/integrations/build/packages/haproxy-1.8.4.zip/kibana/visualization/haproxy-794b6cd0-471d-11e8-bc13-1397384faad3.json" is invalid: found legacy visualization "Active servers in backend [Metrics HAProxy]" (metric, TSVB)

• ibmmq

  • Legacy viz: file "/Users/subhamsarkar/go/src/github.com/elastic/integrations/build/packages/ibmmq-1.2.3.zip/kibana/dashboard/ibmmq-ba1d8830-7c7b-11e9-9645-e37efaf5baff.json" is invalid: "[Logs IBM MQ] Overview of error log" contains legacy visualization: "Top 5 Errors [Logs IBM MQ]" (table, Aggs-based)

• nagios_xi

  • file "/Users/subhamsarkar/go/src/github.com/elastic/integrations/build/packages/nagios_xi-1.1.1.zip/data_stream/service/fields/fields.yml" is invalid: field 0.fields.0.fields.20: Additional property decription is not allowed
    • This is not expected. Needs to be checked. Skipping for now.

[shmsr]

Pulling out mongodb and influxdb becuase they are blocking this PR.

• mongodb: Needs testing to see if any data loss is there (TSDB)
• influxdb: ci failure (read my earlier comments in this PR for more context)

I've cherry-picked the changes to a different branch. See: #8234

[tommyers-elastic]

awesome - no blockers from me. couple of comments still to resolve though. thanks !

[elasticmachine]

Package golang - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=golang

[elasticmachine]

Package hadoop - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=hadoop

[elasticmachine]

Package iis - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=iis

[elasticmachine]

Package jolokia - 0.3.0 containing this change is available at https://epr.elastic.co/search?package=jolokia

[elasticmachine]

Package kafka - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=kafka

[elasticmachine]

Package kafka_log - 1.4.0 containing this change is available at https://epr.elastic.co/search?package=kafka_log

[elasticmachine]

Package memcached - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=memcached

[elasticmachine]

Package microsoft_sqlserver - 2.3.0 containing this change is available at https://epr.elastic.co/search?package=microsoft_sqlserver

[elasticmachine]

Package mysql - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=mysql

[elasticmachine]

Package nginx - 1.16.0 containing this change is available at https://epr.elastic.co/search?package=nginx