Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2.7 optimization fix #934

Open
wants to merge 88 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
88 commits
Select commit Hold shift + click to select a range
84e00b7
[Backport 2.x] add backport github workflow for security-analytics (#95)
opensearch-trigger-bot[bot] Nov 1, 2022
f6e66d5
add findings enabled flag in datasources (#99) (#104)
opensearch-trigger-bot[bot] Nov 3, 2022
dc558fb
update index monitor method to include namedWriteableRegistry for com…
eirsep Nov 3, 2022
9d6e1c5
Rule field validation during IndexRule Action (#80) (#106)
opensearch-trigger-bot[bot] Nov 3, 2022
c415338
Rollover for alerts/findings history indices (#82) (#108)
opensearch-trigger-bot[bot] Nov 3, 2022
ad18056
add owner field in monitor (#110) (#111)
opensearch-trigger-bot[bot] Nov 4, 2022
efcd59f
automate delete backport branch (#113) (#115)
opensearch-trigger-bot[bot] Nov 5, 2022
81a9124
Add 2.4.0.0 releases notes (#114) (#118)
opensearch-trigger-bot[bot] Nov 5, 2022
ee4ab74
security implementation for security-analytics (#78) (#120)
opensearch-trigger-bot[bot] Nov 7, 2022
1211ad2
add mappings and test case (#89) (#125)
opensearch-trigger-bot[bot] Nov 9, 2022
4091ced
removed grouped listener when fetching findings (#124) (#128)
opensearch-trigger-bot[bot] Nov 9, 2022
e11c00e
Creates bucket level monitors for rules containing aggregations (#92)…
opensearch-trigger-bot[bot] Nov 9, 2022
2f9a513
enabled findings in bucket level monitor creation (#131) (#132)
opensearch-trigger-bot[bot] Nov 9, 2022
1e1cb9b
add loopback mappings (#134) (#137)
opensearch-trigger-bot[bot] Nov 9, 2022
1a2a5f8
fixed GetFindings not searching all indices; fixed proper deletion of…
opensearch-trigger-bot[bot] Nov 9, 2022
71ffbd5
mapper topics fix; Index Rule category validation (#123) (#140)
opensearch-trigger-bot[bot] Nov 10, 2022
22c06b0
Enabled creation detector without rules (#143) (#144)
opensearch-trigger-bot[bot] Nov 10, 2022
134539e
added 404s (#146) (#149)
opensearch-trigger-bot[bot] Nov 11, 2022
a8249c8
populate queries field in FindingDTO in case of bucket level monitor …
opensearch-trigger-bot[bot] Nov 11, 2022
eda3cef
fix to support https as param for security integ tests (#156) (#157)
opensearch-trigger-bot[bot] Nov 14, 2022
ce396ba
Monitor execution and finding integration tests (#139) (#168)
opensearch-trigger-bot[bot] Nov 18, 2022
7aa7e36
fix for running windows integration tests (#176) (#177)
opensearch-trigger-bot[bot] Dec 12, 2022
a7f1d62
fix for flaky integration tests (#167) (#180)
opensearch-trigger-bot[bot] Dec 12, 2022
b315523
Add release notes for 2.4.1 (#183) (#184)
opensearch-trigger-bot[bot] Dec 13, 2022
af7f6cc
add failure logs to github workflows (#186) (#193)
opensearch-trigger-bot[bot] Dec 13, 2022
e91f2a2
fixed findingsDto bug (#102) (#200)
opensearch-trigger-bot[bot] Dec 15, 2022
276d4bb
fix vulnerabiity in yaml constructor (#198) (#202)
opensearch-trigger-bot[bot] Dec 17, 2022
7df9be8
fixed aliases being returned in unmapped_index_fields (#147) (#209)
opensearch-trigger-bot[bot] Dec 24, 2022
3b490a1
implement securetransportaction for get alerts and ack alerts and edg…
opensearch-trigger-bot[bot] Dec 24, 2022
ee411e5
search returns detector type in CAPS fix and integration tests (#174)…
opensearch-trigger-bot[bot] Dec 30, 2022
ddbcb21
updated network mappings (#211) (#224)
opensearch-trigger-bot[bot] Jan 4, 2023
ef2c837
update Linux mappings (#223) (#233)
opensearch-trigger-bot[bot] Jan 5, 2023
590d9a4
changes to return empty serach response for custom rules (#231) (#234)
opensearch-trigger-bot[bot] Jan 5, 2023
82c14cd
fix flaky integration tests for security analytics (#241) (#242)
opensearch-trigger-bot[bot] Jan 5, 2023
7fbb882
[Backport 2.x] Added dummy search when creating detector on the given…
opensearch-trigger-bot[bot] Jan 7, 2023
15d3942
updated windows mappings (#212) (#252)
opensearch-trigger-bot[bot] Jan 9, 2023
70f5a09
fix ci builds for security-analytics (#253) (#254)
opensearch-trigger-bot[bot] Jan 9, 2023
153ad3d
[backport 2.x] Removed create/delete queryIndex (#215) (#255)
sbcd90 Jan 10, 2023
a73beb7
GetMappingsView API - index pattern/alias/datastream support (#245) (…
opensearch-trigger-bot[bot] Jan 10, 2023
fb46dc9
Updated MAINTAINERS.md to match recommended opensearch-project format…
opensearch-trigger-bot[bot] Jan 10, 2023
8e76db3
createmappings api index pattern support (#260) (#266)
opensearch-trigger-bot[bot] Jan 11, 2023
516c642
add mapping fixes (#264) (#271)
opensearch-trigger-bot[bot] Jan 11, 2023
5a37a2b
Added 2.5 release notes. (#268) (#269)
opensearch-trigger-bot[bot] Jan 11, 2023
3e8b089
correct linux mapping error (#263) (#273)
opensearch-trigger-bot[bot] Jan 11, 2023
116d08e
GetIndexMappings index pattern support (#265) (#275)
opensearch-trigger-bot[bot] Jan 11, 2023
2c030b2
Service Returns Unhandled Error Response (#248) (#280)
sbcd90 Jan 11, 2023
04f3387
[Backport 2.x] Adds timestamp field alias and sets time range filter …
sbcd90 Jan 11, 2023
4c89ba7
added resolving index template config; changed GetIndexMappings API t…
opensearch-trigger-bot[bot] Jan 11, 2023
3debc83
add nesting level to yaml constructor (#286) (#289)
opensearch-trigger-bot[bot] Jan 11, 2023
120bf78
GetIndexMapping API bugfix (#293) (#294)
opensearch-trigger-bot[bot] Jan 12, 2023
1db129f
log message change for debugging (#321) (#322)
opensearch-trigger-bot[bot] Feb 9, 2023
4f6e58b
Baselined MAINTAINERS and CODEOWNERS docs. (#329)
AWSHurneyt Feb 16, 2023
242bd95
Merge pull request #330 from opensearch-project/backport/backport-329…
AWSHurneyt Feb 16, 2023
dd56ee2
update others_application mappings (#277) (#338)
opensearch-trigger-bot[bot] Feb 20, 2023
de48737
update others_apt (#278) (#339)
opensearch-trigger-bot[bot] Feb 20, 2023
f7d042c
query_field_names bugfix (#335) (#342)
opensearch-trigger-bot[bot] Feb 21, 2023
e5c5c6e
GetAllRuleCategories API (#327) (#343)
opensearch-trigger-bot[bot] Feb 21, 2023
9901e58
update others_cloud mappings (#301) (#344)
opensearch-trigger-bot[bot] Feb 21, 2023
c94b18a
update others_compliance mappings (#302) (#346)
opensearch-trigger-bot[bot] Feb 21, 2023
3444c39
update others_web mappings (#304) (#348)
opensearch-trigger-bot[bot] Feb 21, 2023
591662c
Bump to OpenSearch 2.6 (#351)
lezzago Feb 21, 2023
1eeaadc
Publish snapshots to maven via GHA (#352) (#354)
opensearch-trigger-bot[bot] Feb 21, 2023
2323667
Drafted 2.6 release notes. (#353)
AWSHurneyt Feb 22, 2023
735edf5
Support for multiple indices in detector input (#336) (#356)
opensearch-trigger-bot[bot] Feb 22, 2023
3f09f52
Index template cleanup (#317) (#358)
opensearch-trigger-bot[bot] Feb 22, 2023
d11940a
New log types (#332) (#360)
sbcd90 Feb 22, 2023
c05fa64
fix for integ test failures in 2.6 (#363)
sbcd90 Feb 23, 2023
a4cbdff
2.7 bump (#387)
stevanbz Apr 10, 2023
6974b33
handle index not exists for detector search and delete (#396) (#397)
opensearch-trigger-bot[bot] Apr 12, 2023
2e9dbbd
Handle monitor or monitor index not exists during detector deletion (…
opensearch-trigger-bot[bot] Apr 12, 2023
2d84a6f
Added 2.7 release notes. (#401)
AWSHurneyt Apr 17, 2023
147d784
Merge pull request #402 from opensearch-project/backport/backport-401…
bowenlan-amzn Apr 18, 2023
303f202
add correlation engine for security-analytics (#405) (#407)
opensearch-trigger-bot[bot] Apr 19, 2023
cac3b25
Findings index mappings fix (#409) (#416)
opensearch-trigger-bot[bot] May 2, 2023
4d1525f
enabled source filtering when searching rules (#374) (#420)
opensearch-trigger-bot[bot] May 2, 2023
bf9c3b8
Alias and dataStream end-to-end ITs (#373) (#422)
opensearch-trigger-bot[bot] May 2, 2023
4e360ef
add rules to correlations for correlation engine (#423) (#427)
opensearch-trigger-bot[bot] May 3, 2023
075970a
fix for input validation of correlation rule names (#428) (#431)
opensearch-trigger-bot[bot] May 8, 2023
cd23067
[Backport 2.7] Auto expand replicas (#551)
opensearch-trigger-bot[bot] Sep 13, 2023
830a3b9
* #709 Return empty response for empty mappings and no applied aliase…
goyamegh Dec 14, 2023
0c32858
* changed windows sample rule and query construction (#745)
jowg-amazon Mar 11, 2024
044d6fb
[BUG] ArrayIndexOutOfBoundsException for inconsistent detector index …
opensearch-trigger-bot[bot] Mar 12, 2024
58c33a9
Add throw for empty strings in rules with modifier contains, startwit…
opensearch-trigger-bot[bot] Mar 12, 2024
a94d9c4
Version bump to 2.7.1
goyamegh Mar 14, 2024
c9bb3fd
fix detector writeTo() method missing fields (#695)
eirsep Oct 26, 2023
8dbd4f6
fix null query filter conversion from sigma to query string query (#722)
eirsep Nov 29, 2023
f2ab3e8
* #709 Return empty response for empty mappings and no applied aliases
goyamegh Nov 30, 2023
5fd677a
Pass rule field names in doc level queries during monitor/creation. R…
eirsep Mar 6, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
29 changes: 29 additions & 0 deletions .github/workflows/backport.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,29 @@

name: Backport
on:
pull_request_target:
types:
- closed
- labeled

jobs:
backport:
runs-on: ubuntu-latest
permissions:
contents: write
pull-requests: write
name: Backport
steps:
- name: GitHub App token
id: github_app_token
uses: tibdex/github-app-token@v1.5.0
with:
app_id: ${{ secrets.APP_ID }}
private_key: ${{ secrets.APP_PRIVATE_KEY }}
installation_id: 22958780

- name: Backport
uses: VachaShah/backport@v1.1.4
with:
github_token: ${{ steps.github_app_token.outputs.token }}
branch_name: backport/backport-${{ github.event.number }}
25 changes: 23 additions & 2 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,11 +18,11 @@ jobs:
os: [ ubuntu-latest, windows-latest, macos-latest ]
include:
- os: windows-latest
os_build_args: -x integTest -x jacocoTestReport
os_build_args: -x jacocoTestReport
working_directory: X:\
os_java_options: -Xmx4096M
- os: macos-latest
os_build_args: -x integTest -x jacocoTestReport
os_build_args: -x jacocoTestReport

name: Build and Test security-analytics with JDK ${{ matrix.java }} on ${{ matrix.os }}
runs-on: ${{ matrix.os }}
Expand Down Expand Up @@ -59,6 +59,27 @@ jobs:
with:
token: ${{ secrets.CODECOV_TOKEN }}

- name: Upload failed logs
uses: actions/upload-artifact@v2
if: ${{ failure() && matrix.os == 'ubuntu-latest' }}
with:
name: logs-ubuntu
path: build/testclusters/integTest-*/logs/*

- name: Upload failed logs
uses: actions/upload-artifact@v2
if: ${{ failure() && matrix.os == 'macos-latest' }}
with:
name: logs-mac
path: build/testclusters/integTest-*/logs/*

- name: Upload failed logs
uses: actions/upload-artifact@v2
if: ${{ failure() && matrix.os == 'windows-latest' }}
with:
name: logs-windows
path: build\testclusters\integTest-*\logs\*

- name: Upload Artifacts
uses: actions/upload-artifact@v1
with:
Expand Down
20 changes: 20 additions & 0 deletions .github/workflows/delete_backport_branch.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
##
# Copyright OpenSearch Contributors
# SPDX-License-Identifier: Apache-2.0
##

name: Delete merged branch of the backport PRs
on:
pull_request:
types:
- closed

jobs:
delete-branch:
runs-on: ubuntu-latest
if: startsWith(github.event.pull_request.head.ref,'backport/')
steps:
- name: Delete merged branch
uses: SvanBoxel/delete-merged-branch@main
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
39 changes: 39 additions & 0 deletions .github/workflows/maven-publish.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Publish snapshots to maven

on:
workflow_dispatch:
push:
branches: [
main
1.*
2.*
]

jobs:
build-and-publish-snapshots:
strategy:
fail-fast: false
if: github.repository == 'opensearch-project/security-analytics'
runs-on: ubuntu-latest

permissions:
id-token: write
contents: write

steps:
- uses: actions/setup-java@v3
with:
distribution: temurin # Temurin is a distribution of adoptium
java-version: 17
- uses: actions/checkout@v3
- uses: aws-actions/configure-aws-credentials@v1
with:
role-to-assume: ${{ secrets.PUBLISH_SNAPSHOTS_ROLE }}
aws-region: us-east-1
- name: publish snapshots to maven
run: |
export SONATYPE_USERNAME=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-username --query SecretString --output text)
export SONATYPE_PASSWORD=$(aws secretsmanager get-secret-value --secret-id maven-snapshots-password --query SecretString --output text)
echo "::add-mask::$SONATYPE_USERNAME"
echo "::add-mask::$SONATYPE_PASSWORD"
./gradlew publishPluginZipPublicationToSnapshotsRepository
6 changes: 6 additions & 0 deletions .github/workflows/multi-node-test-workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -28,3 +28,9 @@ jobs:
uses: actions/checkout@v2
- name: Run integration tests with multi node config
run: ./gradlew integTest -PnumNodes=3
- name: Upload failed logs
uses: actions/upload-artifact@v2
if: failure()
with:
name: logs
path: build/testclusters/integTest-*/logs/*
88 changes: 88 additions & 0 deletions .github/workflows/security-test-workflow.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
name: Security Test Workflow
# This workflow is triggered on pull requests and pushes to main or an OpenSearch release branch
on:
pull_request:
branches:
- "*"
push:
branches:
- "*"

jobs:
build:
strategy:
matrix:
java: [ 11, 17 ]
# Job name
name: Build and test SecurityAnalytics
# This job runs on Linux
runs-on: ubuntu-latest
steps:
# This step uses the setup-java Github action: https://github.com/actions/setup-java
- name: Set Up JDK ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}
# This step uses the checkout Github action: https://github.com/actions/checkout
- name: Checkout Branch
uses: actions/checkout@v2
# This step uses the setup-java Github action: https://github.com/actions/setup-java
- name: Set Up JDK ${{ matrix.java }}
uses: actions/setup-java@v1
with:
java-version: ${{ matrix.java }}
- name: Build SecurityAnalytics
# Only assembling since the full build is governed by other workflows
run: ./gradlew assemble

- name: Pull and Run Docker
run: |
plugin=`basename $(ls build/distributions/*.zip)`
list_of_files=`ls`
list_of_all_files=`ls build/distributions/`
version=`echo $plugin|awk -F- '{print $4}'| cut -d. -f 1-3`
plugin_version=`echo $plugin|awk -F- '{print $4}'| cut -d. -f 1-4`
qualifier=`echo $plugin|awk -F- '{print $4}'| cut -d. -f 1-1`
candidate_version=`echo $plugin|awk -F- '{print $5}'| cut -d. -f 1-1`
docker_version=$version

[[ -z $candidate_version ]] && candidate_version=$qualifier && qualifier=""

echo plugin version plugin_version qualifier candidate_version docker_version
echo "($plugin) ($version) ($plugin_version) ($qualifier) ($candidate_version) ($docker_version)"
echo $ls $list_of_all_files

if docker pull opensearchstaging/opensearch:$docker_version
then
echo "FROM opensearchstaging/opensearch:$docker_version" >> Dockerfile
echo "RUN if [ -d /usr/share/opensearch/plugins/opensearch-security-analytics ]; then /usr/share/opensearch/bin/opensearch-plugin remove opensearch-security-analytics; fi" >> Dockerfile
echo "ADD build/distributions/$plugin /tmp/" >> Dockerfile
echo "RUN /usr/share/opensearch/bin/opensearch-plugin install --batch file:/tmp/$plugin" >> Dockerfile

docker build -t opensearch-security-analytics:test .
echo "imagePresent=true" >> $GITHUB_ENV
else
echo "imagePresent=false" >> $GITHUB_ENV
fi

- name: Run Docker Image
if: env.imagePresent == 'true'
run: |
cd ..
docker run -p 9200:9200 -d -p 9600:9600 -e "discovery.type=single-node" opensearch-security-analytics:test
sleep 120

- name: Run SecurityAnalytics Test for security enabled test cases
if: env.imagePresent == 'true'
run: |
cluster_running=`curl -XGET https://localhost:9200/_cat/plugins -u admin:admin --insecure`
echo $cluster_running
security=`curl -XGET https://localhost:9200/_cat/plugins -u admin:admin --insecure |grep opensearch-security|wc -l`
echo $security
if [ $security -gt 0 ]
then
echo "Security plugin is available"
./gradlew :integTest -Dtests.rest.cluster=localhost:9200 -Dtests.cluster=localhost:9200 -Dtests.clustername=docker-cluster -Dhttps=true -Duser=admin -Dpassword=admin
else
echo "Security plugin is NOT available skipping this run as tests without security have already been run"
fi
7 changes: 7 additions & 0 deletions CODEOWNERS
Validating CODEOWNERS rules …
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
@amsiglan
@awshurneyt
@getsaurabh02
@lezzago
@praveensameneni
@sbcd90
@eirsep
20 changes: 13 additions & 7 deletions MAINTAINERS.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,15 @@
## Maintainers
| Maintainer | GitHub ID | Affiliation |
|------------------------|-------------------------------------------------| ----------- |
| Saurabh Singh | [getsaurabh02](https://github.com/getsaurabh02) | Amazon |
| Subhobrata Dey | [sbcd90](https://github.com/sbcd90) | Amazon |
| Surya Sashank Nistalai | [eirsep](https://github.com/eirsep) | Amazon |
## Overview

This document contains a list of maintainers in this repo. See [opensearch-project/.github/RESPONSIBILITIES.md](https://github.com/opensearch-project/.github/blob/main/RESPONSIBILITIES.md#maintainer-responsibilities) that explains what the role of maintainer means, what maintainers do in this and other repos, and how they should be doing it. If you're interested in contributing, and becoming a maintainer, see [CONTRIBUTING](CONTRIBUTING.md).

[This document](https://github.com/opensearch-project/.github/blob/main/MAINTAINERS.md) explains what maintainers do in this repo, and how they should be doing it. If you're interested in contributing, see [CONTRIBUTING](CONTRIBUTING.md).
## Current Maintainers

| Maintainer | GitHub ID | Affiliation |
| ---------------- | ----------------------------------------------------- | ----------- |
| Ashish Agrawal | [lezzago](https://github.com/lezzago) | Amazon |
| Subhobrata Dey | [sbcd90](https://github.com/sbcd90) | Amazon |
| Thomas Hurney | [awshurneyt](https://github.com/AWSHurneyt) | Amazon |
| Surya Sashank Nistala | [eirsep](https://github.com/eirsep) | Amazon |
| Praveen Sameneni | [praveensameneni](https://github.com/praveensameneni) | Amazon |
| Amardeepsingh Siglani | [amsiglan](https://github.com/amsiglan) | Amazon |
| Saurabh Singh | [getsaurabh02](https://github.com/getsaurabh02) | Amazon |
Loading