Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[integrations][Custom GCS] - Converted Custom GCS integration to Input Package type #9725

Merged
merged 37 commits into from May 6, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
37 commits
Select commit Hold shift + click to select a range
d092526
refactored the integration, added new parameters and improved documen…
ShourieG Nov 15, 2023
6aed381
Merge remote-tracking branch 'upstream/main'
ShourieG Nov 16, 2023
043affd
merged with main and resolved merge conflicts
ShourieG Nov 27, 2023
74cd1da
Merge remote-tracking branch 'upstream/main'
ShourieG Nov 29, 2023
e2981a8
Merge remote-tracking branch 'upstream/main'
ShourieG Nov 30, 2023
be9a913
Merge remote-tracking branch 'upstream/main'
ShourieG Dec 6, 2023
301df37
Merge remote-tracking branch 'upstream/main'
ShourieG Dec 7, 2023
b19996c
Merge remote-tracking branch 'upstream/main'
ShourieG Feb 27, 2024
1fb0cbe
Merge remote-tracking branch 'upstream/main'
ShourieG Feb 28, 2024
2832f71
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 1, 2024
aa36ffe
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 7, 2024
6de9853
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 11, 2024
2b7fdb2
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 13, 2024
3208dff
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 18, 2024
23c49a3
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 18, 2024
b4f19e6
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 20, 2024
a601c9f
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 21, 2024
4bab737
Merge remote-tracking branch 'upstream/main'
ShourieG Mar 28, 2024
3305f7d
Merge remote-tracking branch 'upstream/main'
ShourieG Apr 11, 2024
fea5b77
Merge remote-tracking branch 'upstream/main'
ShourieG Apr 12, 2024
82566d3
Merge remote-tracking branch 'upstream/main'
ShourieG Apr 12, 2024
16699cf
Merge remote-tracking branch 'upstream/main'
ShourieG Apr 15, 2024
5f9465e
Merge remote-tracking branch 'upstream/main'
ShourieG Apr 16, 2024
d7af4ef
[Netskope] - Added support for custom tcp options & upgraded stack re…
ShourieG Apr 16, 2024
ab0e9d6
[citrix_adc] Update description to include log processing (#9562)
janvi-elastic Apr 16, 2024
9641969
rebased with upstream/main
ShourieG Apr 16, 2024
2413680
Merge branch 'main' into HEAD
ShourieG Apr 16, 2024
415f35f
Merge remote-tracking branch 'upstream/main'
ShourieG Apr 17, 2024
b50ab23
initial refacor to input package completed with working tests
ShourieG Apr 22, 2024
711a120
updated readme
ShourieG Apr 22, 2024
714fed0
Merge remote-tracking branch 'upstream/main' into ip_migration/gcs
ShourieG Apr 26, 2024
dc880fa
Merge branch 'main' of https://github.com/ShourieG/integrations
ShourieG Apr 26, 2024
e2fcd1b
Merge branch 'main' into ip_migration/gcs
ShourieG Apr 26, 2024
e2c4c5c
updated changelog
ShourieG Apr 27, 2024
3a96b7f
addressed pr comments, updated strings to scalar type
ShourieG Apr 29, 2024
c5378da
added preserve original event option to manifest and updated documant…
ShourieG Apr 30, 2024
1899b95
updated agent.yml to fix tags condition
ShourieG Apr 30, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
57 changes: 0 additions & 57 deletions packages/google_cloud_storage/_dev/build/docs/README.md

This file was deleted.

@@ -0,0 +1,10 @@
service: google-cloud-storage-emulator
input: gcs
vars:
project_id: testproject
alternative_host: "http://{{Hostname}}:{{Port}}"
data_stream.dataset: google_cloud_storage.gcs
buckets: |
- name: testbucket
poll: true
poll_interval: 15s
Expand Up @@ -38,12 +38,13 @@ file_selectors:
{{#if timestamp_epoch}}
timestamp_epoch: {{timestamp_epoch}}
{{/if}}
{{#if tags}}
tags:
{{#if preserve_original_event}}
- preserve_original_event
{{/if}}
{{#each tags as |tag i|}}
- {{tag}}
{{/each}}
{{/if}}
{{#contains "forwarded" tags}}
publisher_pipeline.disable_host: true
{{/contains}}
Expand Down
5 changes: 5 additions & 0 deletions packages/google_cloud_storage/changelog.yml
@@ -1,3 +1,8 @@
- version: "2.0.0"
changes:
- description: Convert Google Cloud Storage to input package type.
type: enhancement
link: https://github.com/elastic/integrations/pull/9725
- version: "1.1.0"
changes:
- description: Set sensitive values as secret.
Expand Down

This file was deleted.

115 changes: 0 additions & 115 deletions packages/google_cloud_storage/data_stream/generic/manifest.yml

This file was deleted.

2 changes: 1 addition & 1 deletion packages/google_cloud_storage/docs/README.md
Expand Up @@ -14,7 +14,7 @@ even though it can get expensive with dealing with a very large number of files.
3. If any major error occurs that stops the main thread, the logs will be appropriately generated, describing said error.


NOTE: Currently only `JSON` is supported with respect to object/file formats. We also support gzipped JSON objects/files. As for authentication types, we currently have support for `JSON credential keys` and `credential files`. If a download for a file/object fails or gets interrupted, the download is retried for 2 times. This is currently not user-configurable.
NOTE: Currently only `JSON` is supported with respect to object/file formats. We also support gzipped JSON objects/files. As for authentication types, we currently have support for `JSON credential keys` and `credential files`. If a download for a file/object fails or gets interrupted, the download is retried two times which equates to a maximum of three tries for a particular file/object. This is behavior currently not user-configurable.

## The GCS credentials key file:
This is a one-time download JSON key file that you get after adding a key to a GCP service account.
Expand Down
35 changes: 35 additions & 0 deletions packages/google_cloud_storage/fields/agent.yml
@@ -0,0 +1,35 @@
- name: cloud
title: Cloud
group: 2
description: Fields related to the cloud or infrastructure the events are coming from.
footnote: 'Examples: If Metricbeat is running on an EC2 host and fetches data from its host, the cloud info contains the data about this machine. If Metricbeat runs on a remote machine outside the cloud and fetches data from a service running in the cloud, the field contains cloud data from the machine the service is running on.'
type: group
fields:
- name: project.id
type: keyword
description: Name of the project in Google Cloud.
- name: image.id
type: keyword
description: Image ID for the cloud instance.
- name: host
title: Host
group: 2
description: 'A host is defined as a general computing instance. ECS host.* fields should be populated with details about the host on which the event happened, or from which the measurement was taken. Host types include hardware, virtual machines, Docker containers, and Kubernetes nodes.'
type: group
fields:
- name: containerized
type: boolean
description: >
If the host is a container.

- name: os.build
type: keyword
example: "18D109"
description: >
OS build information.

- name: os.codename
type: keyword
example: "stretch"
description: >-
OS codename, if any.