Migrate obs-infraobs-integrations to package-spec v3 #3

[shmsr]

Migrate a batch of packages from v1/v2 to v3 package-spec. Used the following scripts and some manual changes to generate the changes in this PR:

yq_hack.zsh

#!/bin/bash

set -e
set -u
set -o pipefail
set -x

pushd $1

find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].release" {} \;
find data_stream -type f -path "*fields*" -name "ecs.yml" -exec yq e -i "unique_by(.name)" {} \;
find data_stream -type f -path "*fields*" -name "*.yml" -exec yq e -i "del .[].fields.[].required" {} \;

yq e -i 'del .release' manifest.yml
yq e -i "with(select(.license != null); .conditions.elastic.subscription = .license) | del .license" manifest.yml

popd

git add -u $1/
git commit -m "[$1]: migration with yq"

ecs-update.zsh

#!/bin/bash

set -u
set -o pipefail
set -x

echo "-- executing ecs-update for $1 --"

ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1

if (( $? )) 
then
    echo "-- write validation for $1 --"
    pushd $1

    go run ../scripting/write_validation.go

    popd

    # Try again?
    git checkout .
    ecs-update -format-version=3.0.0 -fix-dotted-yaml-keys -add-owner-type -owner elastic/obs-infraobs-integrations -v -skip-format=true $1
fi

And another Go program to write the validation.yml by parsing the elastic-package lint errors.

Then I hooked this scripts to run with:

$ pkgs=($(cat .github/CODEOWNERS | grep '@elastic/obs-infraobs-integrations' | cut -d ' ' -f 1 | cut -d '/' -f 3 | sort | uniq))

$ for ((idx=16; idx<=30; idx++))
do
scripting/yq_hack.zsh $pkgs[idx]
done

$ for ((idx=16; idx<=30; idx++))
do
scripting/ecs_update.zsh $pkgs[idx] || break
done

Another variation of the above script:

$ for ((idx=31; idx<=48; idx++))
do
case $pkgs[idx] in
  "tomcat" | "salesforce" | "redis") echo "-- skip $pkgs[idx] --";;
  "websphere_application_server" | "system" | "statsd_input" | "sql_input" | "springboot" | "rabbitmq" | "postgresql" | "prometheus_input" | "oracle_weblogic" | "php_fpm" | "oracle") echo "-- done $pkgs[idx] --";;
  *) scripting/ecs_update.zsh $pkgs[idx] || break;;
esac
done

As I am creating a batch of 18 packages here, I am using a for loop. In case the migration is not possible for some packages, I put it in the first case to skip them.

Proposed commit message

Migrate packages owned by obs-infraobs-integrations but few couldn't be migrated. Marking them as follows:

• oracle
• oracle_weblogic
• php_fpm
• postgresql
• prometheus_input
• rabbitmq
• redis
• redisenterprise
• salesforce
• spring_boot
• sql_input
• statsd_input
• system
• tomcat
• traefik
• vsphere
• websphere_application_server
• zookeeper

Please see the comment #8203 (comment) in this PR itself to understand why some packages couldn't be migrated and also learn about the challenges faced when migrating.

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

• [ ]

How to test this PR locally

Related issues

Relates to #8170
Relates to #8171

Screenshots

[shmsr]

spring_boot ci is failing for some reason. not able to get what's happening. some curl is failing when trying to hit a port where spring_boot is supposed to run; returning "connection refused".

[shmsr]

Pulling out statsd_input and sql_input because of the mapping-related ambiguities. Need to discuss the mappings in more detail and then commit the changes for both packages.

[elasticmachine]

Package oracle - 1.22.0 containing this change is available at https://epr.elastic.co/search?package=oracle

[elasticmachine]

Package oracle_weblogic - 1.3.0 containing this change is available at https://epr.elastic.co/search?package=oracle_weblogic

[elasticmachine]

Package php_fpm - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=php_fpm

[elasticmachine]

Package postgresql - 1.17.0 containing this change is available at https://epr.elastic.co/search?package=postgresql

[elasticmachine]

Package prometheus_input - 0.2.0 containing this change is available at https://epr.elastic.co/search?package=prometheus_input

[elasticmachine]

Package rabbitmq - 1.11.0 containing this change is available at https://epr.elastic.co/search?package=rabbitmq

[elasticmachine]

Package redisenterprise - 0.6.0 containing this change is available at https://epr.elastic.co/search?package=redisenterprise

[elasticmachine]

Package spring_boot - 0.13.0 containing this change is available at https://epr.elastic.co/search?package=spring_boot

[elasticmachine]

Package traefik - 1.11.0 containing this change is available at https://epr.elastic.co/search?package=traefik

[elasticmachine]

Package vsphere - 1.9.0 containing this change is available at https://epr.elastic.co/search?package=vsphere

[elasticmachine]

Package websphere_application_server - 1.1.0 containing this change is available at https://epr.elastic.co/search?package=websphere_application_server

[elasticmachine]

Package zookeeper - 1.10.0 containing this change is available at https://epr.elastic.co/search?package=zookeeper