Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AWS] [EMR] Add logs data stream #6895

Merged
merged 17 commits into from Jul 20, 2023
Merged

[AWS] [EMR] Add logs data stream #6895

merged 17 commits into from Jul 20, 2023

Conversation

gpop63
Copy link
Contributor

@gpop63 gpop63 commented Jul 10, 2023 •
edited

What does this PR do?

Adds EMR logs data stream.

Uses multiline to process logs that span multiple lines.

Checklist

  • I have reviewed tips for building integrations and this pull request is aligned with them.
  • I have verified that all data streams collect metrics or logs.
  • I have added an entry to my package's changelog.yml file.
  • I have verified that Kibana version constraints are current according to guidelines.

Author's Checklist

  • [ ]

How to test this PR locally

Related issues

Screenshots

@gpop63 gpop63 self-assigned this Jul 10, 2023
@elasticmachine
Copy link

elasticmachine commented Jul 10, 2023 •
edited

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS
Pipeline View Test View Changes Artifacts preview preview

Expand to view the summary

Build stats

  • Start Time: 2023-07-20T15:38:33.337+0000

  • Duration: 52 min 54 sec

Test stats 🧪

Test Results
Failed 0
Passed 202
Skipped 4
Total 206

🤖 GitHub comments

Expand to view the GitHub comments

To re-run your PR in the CI, just comment with:

  • /test : Re-trigger the build.

@elasticmachine
Copy link

elasticmachine commented Jul 10, 2023 •
edited

🌐 Coverage report

Name Metrics % (covered/total) Diff
Packages 100.0% (17/17) 💚
Files 94.444% (17/18) 👍 0.327
Classes 94.444% (17/18) 👍 0.327
Methods 85.953% (257/299) 👍 0.091
Lines 86.011% (7501/8721) 👍 0.01
Conditionals 100.0% (0/0) 💚

kaiyan-sheng
kaiyan-sheng reviewed Jul 13, 2023
View reviewed changes
packages/aws/manifest.yml Outdated
description: Collecting logs from EMR using aws-s3 input
input_group: logs
- type: aws-cloudwatch
title: Collect EC2 logs from CloudWatch
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

EMR

packages/aws/data_stream/emr_logs/manifest.yml Outdated
template_path: aws-s3.yml.hbs
title: AWS EMR Logs via S3
description: Collect AWS EMR logs using s3 input
enabled: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would make this one enabled: true since EMR sends logs to S3.

packages/aws/data_stream/emr_logs/manifest.yml Show resolved Hide resolved
packages/aws/data_stream/emr_logs/sample_event.json Show resolved Hide resolved
packages/aws/data_stream/emr_logs/_dev/test/pipeline/test-emr-hadoop.log Outdated
STARTUP_MSG: host = ip-172-31-25-102.eu-central-1.compute.internal/172.31.25.102
STARTUP_MSG: args = [-format, -nonInteractive]
STARTUP_MSG: version = 3.3.3-amzn-3
STARTUP_MSG: classpath = /etc/hadoop/conf:/usr/lib/hadoop/lib/jetty-security-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/accessors-smart-2.4.7.jar:/usr/lib/hadoop/lib/jersey-core-1.19.jar:/usr/lib/hadoop/lib/animal-sniffer-annotations-1.17.jar:/usr/lib/hadoop/lib/nimbus-jose-jwt-9.8.1.jar:/usr/lib/hadoop/lib/asm-5.0.4.jar:/usr/lib/hadoop/lib/jersey-server-1.19.jar:/usr/lib/hadoop/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop/lib/reload4j-1.2.22.jar:/usr/lib/hadoop/lib/avro-1.7.7.jar:/usr/lib/hadoop/lib/jsp-api-2.1.jar:/usr/lib/hadoop/lib/checker-qual-2.5.2.jar:/usr/lib/hadoop/lib/jsr305-3.0.2.jar:/usr/lib/hadoop/lib/commons-beanutils-1.9.4.jar:/usr/lib/hadoop/lib/paranamer-2.3.jar:/usr/lib/hadoop/lib/commons-cli-1.2.jar:/usr/lib/hadoop/lib/jsr311-api-1.1.1.jar:/usr/lib/hadoop/lib/commons-codec-1.15.jar:/usr/lib/hadoop/lib/jersey-json-1.19.jar:/usr/lib/hadoop/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop/lib/jul-to-slf4j-1.7.36.jar:/usr/lib/hadoop/lib/commons-compress-1.21.jar:/usr/lib/hadoop/lib/jersey-servlet-1.19.jar:/usr/lib/hadoop/lib/commons-configuration2-2.8.0.jar:/usr/lib/hadoop/lib/kerb-admin-1.0.1.jar:/usr/lib/hadoop/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop/lib/kerb-client-1.0.1.jar:/usr/lib/hadoop/lib/commons-io-2.8.0.jar:/usr/lib/hadoop/lib/kerb-common-1.0.1.jar:/usr/lib/hadoop/lib/commons-lang3-3.12.0.jar:/usr/lib/hadoop/lib/kerb-core-1.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop/lib/kerb-crypto-1.0.1.jar:/usr/lib/hadoop/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop/lib/commons-net-3.6.jar:/usr/lib/hadoop/lib/kerb-identity-1.0.1.jar:/usr/lib/hadoop/lib/commons-text-1.10.0.jar:/usr/lib/hadoop/lib/kerb-server-1.0.1.jar:/usr/lib/hadoop/lib/curator-client-4.2.0.jar:/usr/lib/hadoop/lib/kerb-simplekdc-1.0.1.jar:/usr/lib/hadoop/lib/curator-framework-4.2.0.jar:/usr/lib/hadoop/lib/kerb-util-1.0.1.jar:/usr/lib/hadoop/lib/curator-recipes-4.2.0.jar:/usr/lib/hadoop/lib/re2j-1.1.jar:/usr/lib/hadoop/lib/dnsjava-2.1.7.jar:/usr/lib/hadoop/lib/kerby-asn1-1.0.1.jar:/usr/lib/hadoop/lib/failureaccess-1.0.jar:/usr/lib/hadoop/lib/slf4j-reload4j-1.7.36.jar:/usr/lib/hadoop/lib/gson-2.9.0.jar:/usr/lib/hadoop/lib/slf4j-api-1.7.36.jar:/usr/lib/hadoop/lib/guava-27.0-jre.jar:/usr/lib/hadoop/lib/kerby-config-1.0.1.jar:/usr/lib/hadoop/lib/httpclient-4.5.9.jar:/usr/lib/hadoop/lib/snappy-java-1.1.8.2.jar:/usr/lib/hadoop/lib/httpcore-4.4.11.jar:/usr/lib/hadoop/lib/kerby-pkix-1.0.1.jar:/usr/lib/hadoop/lib/j2objc-annotations-1.1.jar:/usr/lib/hadoop/lib/jettison-1.5.1.jar:/usr/lib/hadoop/lib/jackson-annotations-2.12.7.jar:/usr/lib/hadoop/lib/kerby-util-1.0.1.jar:/usr/lib/hadoop/lib/jackson-core-2.12.7.jar:/usr/lib/hadoop/lib/kerby-xdr-1.0.1.jar:/usr/lib/hadoop/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop/lib/jetty-io-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/jackson-databind-2.12.7.1.jar:/usr/lib/hadoop/lib/metrics-core-3.2.4.jar:/usr/lib/hadoop/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop/lib/jetty-http-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop/lib/netty-all-4.1.77.Final.jar:/usr/lib/hadoop/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop/lib/jetty-webapp-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/jakarta.activation-api-1.2.1.jar:/usr/lib/hadoop/lib/netty-3.10.6.Final.jar:/usr/lib/hadoop/lib/javax.servlet-api-3.1.0.jar:/usr/lib/hadoop/lib/token-provider-1.0.1.jar:/usr/lib/hadoop/lib/jaxb-api-2.2.11.jar:/usr/lib/hadoop/lib/netty-codec-4.1.77.Final.jar:/usr/lib/hadoop/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop/lib/netty-buffer-4.1.77.Final.jar:/usr/lib/hadoop/lib/jcip-annotations-1.0-1.jar:/usr/lib/hadoop/lib/jetty-server-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/jetty-xml-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/jetty-servlet-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/jsch-0.1.55.jar:/usr/lib/hadoop/lib/jetty-util-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/stax2-api-4.2.1.jar:/usr/lib/hadoop/lib/jetty-util-ajax-9.4.48.v20220622.jar:/usr/lib/hadoop/lib/json-smart-2.4.7.jar:/usr/lib/hadoop/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/lib/hadoop/lib/netty-codec-dns-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-haproxy-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-http-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-http2-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-memcache-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-mqtt-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-redis-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-smtp-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-socks-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-stomp-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-codec-xml-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-common-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-handler-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-handler-proxy-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-resolver-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-resolver-dns-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-resolver-dns-classes-macos-4.1.77.Final.jar:/usr/lib/hadoop/lib/zookeeper-d*.jar:/usr/lib/hadoop/lib/netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar:/usr/lib/hadoop/lib/netty-resolver-dns-native-macos-4.1.77.Final-osx-x86_64.jar:/usr/lib/hadoop/lib/netty-transport-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-transport-classes-epoll-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-transport-classes-kqueue-4.1.77.Final.jar:/usr/lib/hadoop/lib/zookeeper-jute-3.5.10.jar:/usr/lib/hadoop/lib/netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar:/usr/lib/hadoop/lib/netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:/usr/lib/hadoop/lib/netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar:/usr/lib/hadoop/lib/netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar:/usr/lib/hadoop/lib/netty-transport-native-unix-common-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-transport-rxtx-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-transport-sctp-4.1.77.Final.jar:/usr/lib/hadoop/lib/netty-transport-udt-4.1.77.Final.jar:/usr/lib/hadoop/lib/woodstox-core-5.4.0.jar:/usr/lib/hadoop/lib/zookeeper-3.5.10.jar:/usr/lib/hadoop/.//hadoop-nfs-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-aliyun-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-aliyun.jar:/usr/lib/hadoop/.//hadoop-openstack-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-annotations-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-annotations.jar:/usr/lib/hadoop/.//hadoop-kms-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-archive-logs-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-archive-logs.jar:/usr/lib/hadoop/.//hadoop-nfs.jar:/usr/lib/hadoop/.//hadoop-archives-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-archives.jar:/usr/lib/hadoop/.//hadoop-openstack.jar:/usr/lib/hadoop/.//hadoop-auth-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-auth.jar:/usr/lib/hadoop/.//hadoop-aws-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-aws.jar:/usr/lib/hadoop/.//hadoop-registry-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-azure-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-kms.jar:/usr/lib/hadoop/.//hadoop-azure-datalake-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-azure-datalake.jar:/usr/lib/hadoop/.//hadoop-azure.jar:/usr/lib/hadoop/.//hadoop-registry.jar:/usr/lib/hadoop/.//hadoop-client-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-client.jar:/usr/lib/hadoop/.//hadoop-minicluster-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-common-3.3.3-amzn-3-tests.jar:/usr/lib/hadoop/.//hadoop-shaded-protobuf_3_7-1.1.1.jar:/usr/lib/hadoop/.//hadoop-common-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-common.jar:/usr/lib/hadoop/.//hadoop-sls-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-datajoin-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-datajoin.jar:/usr/lib/hadoop/.//hadoop-sls.jar:/usr/lib/hadoop/.//hadoop-distcp-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-distcp.jar:/usr/lib/hadoop/.//hadoop-kafka-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-dynamometer-blockgen-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-streaming-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-dynamometer-blockgen.jar:/usr/lib/hadoop/.//hadoop-minicluster.jar:/usr/lib/hadoop/.//hadoop-dynamometer-infra-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-streaming.jar:/usr/lib/hadoop/.//hadoop-dynamometer-infra.jar:/usr/lib/hadoop/.//hadoop-kafka.jar:/usr/lib/hadoop/.//hadoop-dynamometer-workload-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-dynamometer-workload.jar:/usr/lib/hadoop/.//hadoop-extras-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-extras.jar:/usr/lib/hadoop/.//hadoop-fs2img-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-fs2img.jar:/usr/lib/hadoop/.//hadoop-gridmix-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-gridmix.jar:/usr/lib/hadoop/.//hadoop-shaded-guava-1.1.1.jar:/usr/lib/hadoop/.//hadoop-resourceestimator-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-resourceestimator.jar:/usr/lib/hadoop/.//hadoop-rumen-3.3.3-amzn-3.jar:/usr/lib/hadoop/.//hadoop-rumen.jar:/usr/lib/hadoop-hdfs/./:/usr/lib/hadoop-hdfs/lib/jetty-security-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/accessors-smart-2.4.7.jar:/usr/lib/hadoop-hdfs/lib/jersey-core-1.19.jar:/usr/lib/hadoop-hdfs/lib/animal-sniffer-annotations-1.17.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/asm-5.0.4.jar:/usr/lib/hadoop-hdfs/lib/jersey-server-1.19.jar:/usr/lib/hadoop-hdfs/lib/audience-annotations-0.5.0.jar:/usr/lib/hadoop-hdfs/lib/nimbus-jose-jwt-9.8.1.jar:/usr/lib/hadoop-hdfs/lib/avro-1.7.7.jar:/usr/lib/hadoop-hdfs/lib/json-smart-2.4.7.jar:/usr/lib/hadoop-hdfs/lib/checker-qual-2.5.2.jar:/usr/lib/hadoop-hdfs/lib/jsr305-3.0.2.jar:/usr/lib/hadoop-hdfs/lib/commons-beanutils-1.9.4.jar:/usr/lib/hadoop-hdfs/lib/okio-2.8.0.jar:/usr/lib/hadoop-hdfs/lib/commons-cli-1.2.jar:/usr/lib/hadoop-hdfs/lib/jsr311-api-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/commons-codec-1.15.jar:/usr/lib/hadoop-hdfs/lib/jersey-json-1.19.jar:/usr/lib/hadoop-hdfs/lib/commons-collections-3.2.2.jar:/usr/lib/hadoop-hdfs/lib/kerb-admin-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/commons-compress-1.21.jar:/usr/lib/hadoop-hdfs/lib/jersey-servlet-1.19.jar:/usr/lib/hadoop-hdfs/lib/commons-configuration2-2.8.0.jar:/usr/lib/hadoop-hdfs/lib/kerb-client-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/commons-daemon-1.0.13.jar:/usr/lib/hadoop-hdfs/lib/kerb-common-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/commons-io-2.8.0.jar:/usr/lib/hadoop-hdfs/lib/kerb-core-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/commons-lang3-3.12.0.jar:/usr/lib/hadoop-hdfs/lib/kerb-crypto-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/commons-logging-1.1.3.jar:/usr/lib/hadoop-hdfs/lib/kerb-identity-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/commons-math3-3.1.1.jar:/usr/lib/hadoop-hdfs/lib/okhttp-4.9.3.jar:/usr/lib/hadoop-hdfs/lib/commons-net-3.6.jar:/usr/lib/hadoop-hdfs/lib/kerb-server-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/commons-text-1.10.0.jar:/usr/lib/hadoop-hdfs/lib/kerb-simplekdc-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/curator-client-4.2.0.jar:/usr/lib/hadoop-hdfs/lib/kerb-util-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/curator-framework-4.2.0.jar:/usr/lib/hadoop-hdfs/lib/kerby-asn1-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/curator-recipes-4.2.0.jar:/usr/lib/hadoop-hdfs/lib/paranamer-2.3.jar:/usr/lib/hadoop-hdfs/lib/dnsjava-2.1.7.jar:/usr/lib/hadoop-hdfs/lib/kerby-config-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/failureaccess-1.0.jar:/usr/lib/hadoop-hdfs/lib/reload4j-1.2.22.jar:/usr/lib/hadoop-hdfs/lib/gson-2.9.0.jar:/usr/lib/hadoop-hdfs/lib/protobuf-java-2.5.0.jar:/usr/lib/hadoop-hdfs/lib/guava-27.0-jre.jar:/usr/lib/hadoop-hdfs/lib/kerby-pkix-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/httpclient-4.5.9.jar:/usr/lib/hadoop-hdfs/lib/re2j-1.1.jar:/usr/lib/hadoop-hdfs/lib/httpcore-4.4.11.jar:/usr/lib/hadoop-hdfs/lib/kerby-util-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/j2objc-annotations-1.1.jar:/usr/lib/hadoop-hdfs/lib/jettison-1.5.1.jar:/usr/lib/hadoop-hdfs/lib/jackson-annotations-2.12.7.jar:/usr/lib/hadoop-hdfs/lib/kerby-xdr-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-2.12.7.jar:/usr/lib/hadoop-hdfs/lib/kotlin-stdlib-1.4.10.jar:/usr/lib/hadoop-hdfs/lib/jackson-core-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-io-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/jackson-databind-2.12.7.1.jar:/usr/lib/hadoop-hdfs/lib/kotlin-stdlib-common-1.4.10.jar:/usr/lib/hadoop-hdfs/lib/jackson-jaxrs-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-http-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/jackson-mapper-asl-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/leveldbjni-all-1.8.jar:/usr/lib/hadoop-hdfs/lib/jackson-xc-1.9.13.jar:/usr/lib/hadoop-hdfs/lib/jetty-webapp-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/jakarta.activation-api-1.2.1.jar:/usr/lib/hadoop-hdfs/lib/netty-3.10.6.Final.jar:/usr/lib/hadoop-hdfs/lib/javax.servlet-api-3.1.0.jar:/usr/lib/hadoop-hdfs/lib/snappy-java-1.1.8.2.jar:/usr/lib/hadoop-hdfs/lib/jaxb-api-2.2.11.jar:/usr/lib/hadoop-hdfs/lib/netty-all-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/jaxb-impl-2.2.3-1.jar:/usr/lib/hadoop-hdfs/lib/netty-buffer-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/jcip-annotations-1.0-1.jar:/usr/lib/hadoop-hdfs/lib/jetty-server-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/jetty-xml-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/jetty-servlet-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/jsch-0.1.55.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/netty-resolver-dns-classes-macos-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/jetty-util-ajax-9.4.48.v20220622.jar:/usr/lib/hadoop-hdfs/lib/json-simple-1.1.1.jar:/usr/lib/hadoop-hdfs/lib/listenablefuture-9999.0-empty-to-avoid-conflict-with-guava.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-dns-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-haproxy-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-http-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-http2-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-memcache-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-mqtt-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-redis-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-smtp-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-socks-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-stomp-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-codec-xml-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-common-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-handler-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-handler-proxy-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-resolver-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-resolver-dns-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-resolver-dns-native-macos-4.1.77.Final-osx-aarch_64.jar:/usr/lib/hadoop-hdfs/lib/netty-resolver-dns-native-macos-4.1.77.Final-osx-x86_64.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-classes-epoll-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-classes-kqueue-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-native-epoll-4.1.77.Final-linux-aarch_64.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-native-epoll-4.1.77.Final-linux-x86_64.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-native-kqueue-4.1.77.Final-osx-aarch_64.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-native-kqueue-4.1.77.Final-osx-x86_64.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-native-unix-common-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-rxtx-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-sctp-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/netty-transport-udt-4.1.77.Final.jar:/usr/lib/hadoop-hdfs/lib/stax2-api-4.2.1.jar:/usr/lib/hadoop-hdfs/lib/token-provider-1.0.1.jar:/usr/lib/hadoop-hdfs/lib/woodstox-core-5.4.0.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-3.5.10.jar:/usr/lib/hadoop-hdfs/lib/zookeeper-jute-3.5.10.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-3.3.3-amzn-3-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-3.3.3-amzn-3.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-client-3.3.3-amzn-3-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-client-3.3.3-amzn-3.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-client.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-httpfs-3.3.3-amzn-3.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-httpfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-native-client-3.3.3-amzn-3-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-native-client-3.3.3-amzn-3.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-native-client.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs-3.3.3-amzn-3.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-nfs.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-rbf-3.3.3-amzn-3-tests.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-rbf-3.3.3-amzn-3.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs-rbf.jar:/usr/lib/hadoop-hdfs/.//hadoop-hdfs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common.jar:/usr/lib/hadoop-mapreduce/.//aliyun-java-sdk-core-4.5.10.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs.jar:/usr/lib/hadoop-mapreduce/.//aliyun-java-sdk-kms-2.11.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient.jar:/usr/lib/hadoop-mapreduce/.//aliyun-java-sdk-ram-3.1.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//aliyun-sdk-oss-3.13.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-nativetask.jar:/usr/lib/hadoop-mapreduce/.//azure-data-lake-store-sdk-2.3.9.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle.jar:/usr/lib/hadoop-mapreduce/.//azure-keyvault-core-1.0.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-streaming.jar:/usr/lib/hadoop-mapreduce/.//azure-storage-7.0.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-uploader.jar:/usr/lib/hadoop-mapreduce/.//hadoop-aliyun-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//jdom2-2.0.6.jar:/usr/lib/hadoop-mapreduce/.//hadoop-aliyun.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archive-logs-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hamcrest-core-1.3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archive-logs.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//junit-4.13.2.jar:/usr/lib/hadoop-mapreduce/.//hadoop-archives.jar:/usr/lib/hadoop-mapreduce/.//hadoop-aws-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//kafka-clients-2.8.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-aws.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-examples.jar:/usr/lib/hadoop-mapreduce/.//hadoop-azure-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-app.jar:/usr/lib/hadoop-mapreduce/.//hadoop-azure-datalake-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//jdk.tools-1.8.jar:/usr/lib/hadoop-mapreduce/.//hadoop-azure-datalake.jar:/usr/lib/hadoop-mapreduce/.//ojalgo-43.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-azure.jar:/usr/lib/hadoop-mapreduce/.//hadoop-minicluster-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-client-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//lz4-java-1.7.1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-client.jar:/usr/lib/hadoop-mapreduce/.//hadoop-minicluster.jar:/usr/lib/hadoop-mapreduce/.//hadoop-datajoin-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//opentracing-noop-0.33.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-datajoin.jar:/usr/lib/hadoop-mapreduce/.//hadoop-openstack-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//opentracing-api-0.33.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-distcp.jar:/usr/lib/hadoop-mapreduce/.//hadoop-kafka-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-dynamometer-blockgen-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-openstack.jar:/usr/lib/hadoop-mapreduce/.//hadoop-dynamometer-blockgen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-common-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-dynamometer-infra-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-resourceestimator-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-dynamometer-infra.jar:/usr/lib/hadoop-mapreduce/.//hadoop-kafka.jar:/usr/lib/hadoop-mapreduce/.//opentracing-util-0.33.0.jar:/usr/lib/hadoop-mapreduce/.//hadoop-dynamometer-workload-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-resourceestimator.jar:/usr/lib/hadoop-mapreduce/.//hadoop-dynamometer-workload.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-extras.jar:/usr/lib/hadoop-mapreduce/.//hadoop-rumen.jar:/usr/lib/hadoop-mapreduce/.//hadoop-fs2img-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//zstd-jni-1.4.9-1.jar:/usr/lib/hadoop-mapreduce/.//hadoop-fs2img.jar:/usr/lib/hadoop-mapreduce/.//hadoop-sls-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-gridmix-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-gridmix.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-sls.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-core.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-plugins.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-hs-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//ini4j-0.5.4.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-3.3.3-amzn-3-tests.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-jobclient-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-nativetask-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-shuffle-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//hadoop-mapreduce-client-uploader-3.3.3-amzn-3.jar:/usr/lib/hadoop-mapreduce/.//org.jacoco.agent-0.8.5-runtime.jar:/usr/lib/hadoop-mapreduce/.//wildfly-openssl-1.0.7.Final.jar:/usr/lib/hadoop-yarn/lib/HikariCP-java7-2.4.12.jar:/usr/lib/hadoop-yarn/lib/aopalliance-1.0.jar:/usr/lib/hadoop-yarn/lib/asm-analysis-9.3.jar:/usr/lib/hadoop-yarn/lib/asm-commons-9.3.jar:/usr/lib/hadoop-yarn/lib/asm-tree-9.3.jar:/usr/lib/hadoop-yarn/lib/bcpkix-jdk15on-1.60.jar:/usr/lib/hadoop-yarn/lib/bcprov-jdk15on-1.60.jar:/usr/lib/hadoop-yarn/lib/ehcache-3.3.1.jar:/usr/lib/hadoop-yarn/lib/fst-2.50.jar:/usr/lib/hadoop-yarn/lib/websocket-servlet-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/geronimo-jcache_1.0_spec-1.0-alpha-1.jar:/usr/lib/hadoop-yarn/lib/guice-4.0.jar:/usr/lib/hadoop-yarn/lib/guice-servlet-4.0.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-base-2.12.7.jar:/usr/lib/hadoop-yarn/lib/jackson-jaxrs-json-provider-2.12.7.jar:/usr/lib/hadoop-yarn/lib/jackson-module-jaxb-annotations-2.12.7.jar:/usr/lib/hadoop-yarn/lib/jakarta.xml.bind-api-2.3.2.jar:/usr/lib/hadoop-yarn/lib/java-util-1.9.0.jar:/usr/lib/hadoop-yarn/lib/javax-websocket-client-impl-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/websocket-server-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/javax-websocket-server-impl-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/javax.inject-1.jar:/usr/lib/hadoop-yarn/lib/javax.websocket-api-1.0.jar:/usr/lib/hadoop-yarn/lib/javax.websocket-client-api-1.0.jar:/usr/lib/hadoop-yarn/lib/jersey-client-1.19.jar:/usr/lib/hadoop-yarn/lib/jersey-guice-1.19.jar:/usr/lib/hadoop-yarn/lib/jetty-annotations-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/jetty-client-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/jetty-jndi-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/jetty-plus-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/jline-3.9.0.jar:/usr/lib/hadoop-yarn/lib/jna-5.2.0.jar:/usr/lib/hadoop-yarn/lib/json-io-2.5.1.jar:/usr/lib/hadoop-yarn/lib/metrics-core-3.2.4.jar:/usr/lib/hadoop-yarn/lib/mssql-jdbc-6.2.1.jre7.jar:/usr/lib/hadoop-yarn/lib/objenesis-2.6.jar:/usr/lib/hadoop-yarn/lib/snakeyaml-1.33.jar:/usr/lib/hadoop-yarn/lib/swagger-annotations-1.5.4.jar:/usr/lib/hadoop-yarn/lib/websocket-api-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/websocket-client-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/lib/websocket-common-9.4.48.v20220622.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-services-core-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-distributedshell.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-mawo-core-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-mawo-core.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-applications-unmanaged-am-launcher.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-client.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-registry-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-registry.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-web-proxy-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-applicationhistoryservice.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-common.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-nodemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-resourcemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-router-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-router.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-services-api-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-sharedcachemanager.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-tests.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-services-api.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-timeline-pluginstorage-3.3.3-amzn-3.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-server-timeline-pluginstorage.jar:/usr/lib/hadoop-yarn/.//hadoop-yarn-services-core.jar:/etc/tez/conf:/usr/lib/tez/hadoop-shim-0.10.2-amzn-2.jar:/usr/lib/tez/hadoop-shim-2.8-0.10.2-amzn-2.jar:/usr/lib/tez/lib:/usr/lib/tez/LICENSE:/usr/lib/tez/LICENSE-BSD-3clause:/usr/lib/tez/LICENSE-CDDLv1.1-GPLv2_withCPE:/usr/lib/tez/LICENSE-MIT:/usr/lib/tez/LICENSE-SIL_OpenFontLicense-v1.1:/usr/lib/tez/NOTICE:/usr/lib/tez/tez-api-0.10.2-amzn-2.jar:/usr/lib/tez/tez-aux-services-0.10.2-amzn-2.jar:/usr/lib/tez/tez-aux-services.jar:/usr/lib/tez/tez-build-tools-0.10.2-amzn-2.jar:/usr/lib/tez/tez-common-0.10.2-amzn-2.jar:/usr/lib/tez/tez-dag-0.10.2-amzn-2.jar:/usr/lib/tez/tez-examples-0.10.2-amzn-2.jar:/usr/lib/tez/tez-history-parser-0.10.2-amzn-2.jar:/usr/lib/tez/tez-javadoc-tools-0.10.2-amzn-2.jar:/usr/lib/tez/tez-job-analyzer-0.10.2-amzn-2.jar:/usr/lib/tez/tez-mapreduce-0.10.2-amzn-2.jar:/usr/lib/tez/tez-protobuf-history-plugin-0.10.2-amzn-2.jar:/usr/lib/tez/tez-runtime-internals-0.10.2-amzn-2.jar:/usr/lib/tez/tez-runtime-library-0.10.2-amzn-2.jar:/usr/lib/tez/tez-tests-0.10.2-amzn-2.jar:/usr/lib/tez/tez-ui-0.10.2-amzn-2.war:/usr/lib/tez/tez-yarn-timeline-cache-plugin-0.10.2-amzn-2.jar:/usr/lib/tez/tez-yarn-timeline-history-0.10.2-amzn-2.jar:/usr/lib/tez/tez-yarn-timeline-history-with-acls-0.10.2-amzn-2.jar:/usr/lib/tez/tez-yarn-timeline-history-with-fs-0.10.2-amzn-2.jar:/usr/lib/tez/lib/async-http-client-2.12.3.jar:/usr/lib/tez/lib/async-http-client-netty-utils-2.12.3.jar:/usr/lib/tez/lib/commons-cli-1.2.jar:/usr/lib/tez/lib/commons-codec-1.11.jar:/usr/lib/tez/lib/commons-collections4-4.1.jar:/usr/lib/tez/lib/commons-io-2.8.0.jar:/usr/lib/tez/lib/commons-lang-2.6.jar:/usr/lib/tez/lib/guava-31.1-jre.jar:/usr/lib/tez/lib/hadoop-annotations.jar:/usr/lib/tez/lib/hadoop-auth.jar:/usr/lib/tez/lib/hadoop-hdfs-client-3.3.3-amzn-3.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common-3.3.3-amzn-3.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-common.jar:/usr/lib/tez/lib/hadoop-mapreduce-client-core-3.3.3-amzn-3.jar:/usr/lib/tez/lib/hadoop-yarn-server-timeline-pluginstorage-3.3.3-amzn-3.jar:/usr/lib/tez/lib/hadoop-yarn-server-web-proxy.jar:/usr/lib/tez/lib/hhadoop-mapreduce-client-core.jar:/usr/lib/tez/lib/javax.servlet-api-3.1.0.jar:/usr/lib/tez/lib/jersey-client-1.19.jar:/usr/lib/tez/lib/jersey-json-1.19.jar:/usr/lib/tez/lib/jettison-1.3.4.jar:/usr/lib/tez/lib/jsr305-3.0.0.jar:/usr/lib/tez/lib/metrics-core-3.1.0.jar:/usr/lib/tez/lib/netty-all-4.1.72.Final.jar:/usr/lib/tez/lib/netty-reactive-streams-2.0.4.jar:/usr/lib/tez/lib/protobuf-java-2.5.0.jar:/usr/lib/tez/lib/reactive-streams-1.0.3.jar:/usr/lib/tez/lib/RoaringBitmap-0.7.45.jar:/usr/lib/tez/lib/slf4j-api-1.7.36.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo-0.4.19.jar:/usr/lib/hadoop-lzo/lib/hadoop-lzo.jar:/usr/lib/hadoop-lzo/lib/native:/usr/share/aws/aws-java-sdk/aws-java-sdk-bundle-1.12.446.jar:/usr/share/aws/aws-java-sdk/LICENSE.txt:/usr/share/aws/aws-java-sdk/NOTICE.txt:/usr/share/aws/aws-java-sdk/README.md:/usr/share/aws/emr/emrfs/conf:/usr/share/aws/emr/emrfs/lib/animal-sniffer-annotations-1.14.jar:/usr/share/aws/emr/emrfs/lib/annotations-16.0.2.jar:/usr/share/aws/emr/emrfs/lib/aopalliance-1.0.jar:/usr/share/aws/emr/emrfs/lib/bcprov-ext-jdk15on-1.66.jar:/usr/share/aws/emr/emrfs/lib/checker-qual-2.5.2.jar:/usr/share/aws/emr/emrfs/lib/emrfs-hadoop-assembly-2.56.0.jar:/usr/share/aws/emr/emrfs/lib/error_prone_annotations-2.1.3.jar:/usr/share/aws/emr/emrfs/lib/findbugs-annotations-3.0.1.jar:/usr/share/aws/emr/emrfs/lib/ion-java-1.0.2.jar:/usr/share/aws/emr/emrfs/lib/j2objc-annotations-1.1.jar:/usr/share/aws/emr/emrfs/lib/javax.inject-1.jar:/usr/share/aws/emr/emrfs/lib/jmespath-java-1.12.446.jar:/usr/share/aws/emr/emrfs/lib/jsr305-3.0.2.jar:/usr/share/aws/emr/emrfs/auxlib/*:/usr/share/aws/emr/ddb/lib/emr-ddb-hadoop.jar:/usr/share/aws/emr/goodies/lib/emr-hadoop-goodies.jar:/usr/share/aws/emr/kinesis/lib/emr-kinesis-hadoop.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink-2.5.0.jar:/usr/share/aws/emr/cloudwatch-sink/lib/cloudwatch-sink.jar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

WDYT about making classpath = just a bit shorter? Just for readability of the log 😬

@gpop63 gpop63 marked this pull request as ready for review July 17, 2023 21:52
@gpop63 gpop63 requested a review from a team as a code owner July 17, 2023 21:52
kaiyan-sheng
kaiyan-sheng reviewed Jul 17, 2023
View reviewed changes
Copy link
Contributor

@kaiyan-sheng kaiyan-sheng left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall it looks good to me, just one more thing here: we are adding standalone s3 option to all log data streams that uses aws-s3 input. So we should add it here too. Please see #6915 as an example. Thanks!

@gpop63 gpop63 merged commit d9ea934 into elastic:main Jul 20, 2023
4 checks passed
@elasticmachine
Copy link

Package aws - 1.50.0 containing this change is available at https://epr.elastic.co/search?package=aws

gizas pushed a commit that referenced this pull request Sep 5, 2023
@gpop63 @gizas
[AWS] [EMR] Add logs data stream (#6895)
4647534 
* add emr logs data stream

* fix sample_event

* make pipeline log classpath shorter

* make s3 enabled and cloudwatch disabled by default

* replace EC2 with EMR in manifest

* fix sample_event

* add aws s3 fields

* add aws s3 standalone options

* add logs in docs

* fix config order

* remove redundant field

* better fields naming
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kaiyan-sheng kaiyan-sheng kaiyan-sheng approved these changes

Assignees

@gpop63 gpop63

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[AWS] [EMR] Logs Integration
3 participants
@gpop63 @elasticmachine @kaiyan-sheng