Gangams/logs 50k eps per node #1235

ganga1980 · 2024-04-29T04:35:41Z

This PR has following changes

AMA core agent integration on high log scale mode
Telemetry related to AMA core agent traces and high log scale mode
Extend Config Map for high log scale mode
Use high log scale config related to fluent-bit, such as threading, storage type
Extend liveness probe for AMA Core Agent when high log scale enabled
ARM template updates related to high log scale

Onboarding steps

Configure enabled = true under [agent_settings.high_log_scale] in ConfigMap
Specify Microsoft-ContainerLogV2-HighScale instead of Microsoft-ContainerLogV2 in ARM template parameter file

Scale tests has been performed on VM SKU - Standard_D16s_v5

For 50k eps, we would need minimum 2GB and 3Core.

Validated following scenarios with high log scale mode

ContainerLogV2 with Microsoft-ContainerLogV2-HighScale stream on both Linux and Windows
Here are memory usage numbers for MDSD + AMA Core Agent on various log scales
- No scale MDSD (90MB) and AMACA (90MB)
- 3K eps MDSD(120MB) and AMACA(300MB)
- 50K eps MDSD(120MB) and AMACA(400MB)
HTTP proxy
AMPLS
Both Windows & Linux

Without high log scale mode (i.e. default )

Validated on ARM64, Azure Linux and Windows nodes

Not supported high log scale scenarios -

ARM64
HTTP proxy with CA cert

Resource Usage at 50K eps - CPU is around 4 core and memory is 2.5GB, Disk IOPS - 570.

ganga1980 · 2024-05-07T05:23:26Z

Disk IO charts at 50K logs/sec/node.

Total Max Read & Writes/sec is 570 at peak. At peak, Read/sec is 329 and writes/sec is 242.

ganga1980 · 2024-05-07T05:46:19Z

latest test run with MDSD & AMACA RC image -

ganga1980 · 2024-05-09T01:11:33Z

On default scenario (without high log scale mode) on MSI mode on x64, ARM64 and windows node -

Daemonset Memory Usage (with & without AMA version update)

Daemonset CPU Usage (with & without AMA version update)

Replicaset CPU Usage

Replicaset Memory Usage

On default scenario (without high log scale mode) on Legacy mode on x64, ARM64 and windows node -

Daemonset Memory Usage (with & without AMA version update)

Deamonset CPU Usage (with & without AMA version update)

Replicaset CPU Usage (with & without AMA version update)

Replicaset memory usage(with & without AMA version update)

build/linux/installer/conf/fluent-bit-common.conf

build/linux/installer/scripts/dcr-config-parser.rb

build/common/installer/scripts/fluent-bit-conf-customizer.rb

source/plugins/go/src/extension/extension.go

kubernetes/linux/main.sh

* fix version in Geneva config xml (#1227) * fix bugs (#1230) * fix bugs * fix comment * update dcr optimization error messages (#1228) * update dcr optimization error messages * add additional check for geneva * redirect dcr parser stderr and stdout to traces file --------- Co-authored-by: Amol Agrawal <amagraw@microsoft.com> * update fluent-bit to 2.2.2 in linux (#1229) * update fluent-bit to 2.2.2 in linux --------- Co-authored-by: Amol Agrawal <amagraw@microsoft.com> * update charts, yaml and release notes for 3.1.20 (#1234) Co-authored-by: Amol Agrawal <amagraw@microsoft.com> * Geneva -send windows container inventory and perf with RS (#1233) * Update the geneva feature flag for RS --------- Co-authored-by: Janvi Jatakia (from Dev Box) <jajataki@microsoft.com> * Add scan tools to the build pipeline (#1237) * Add the missing tools to the build pipeline * update policheck similar to prom metrics * update binskim * update trivyignore * add policheck in windows section --------- Co-authored-by: Janvi Jatakia (from Dev Box) <jajataki@microsoft.com> * streamline input plugin code. (#1238) * streamline input plugin code --------- Co-authored-by: Amol Agrawal <amagraw@microsoft.com> * Telemetry optimization: adding addon token adapter traces as metrics (#1231) * Add token adapter traces as metrics * update trivyignore * updating name of mdsd function * Updating the addon token adapter to discard unnecessary logs * Update trivyignore --------- Co-authored-by: Janvi Jatakia (from Dev Box) <jajataki@microsoft.com> * Update ai instrumentation key for USNAT/USSEC (#1239) * update ai instrumentation key * address comments * resolve comments * syntax error --------- Co-authored-by: Janvi Jatakia (from Dev Box) <jajataki@microsoft.com> * Gangams/logs 50k eps per node (#1235) * mdsd version 50k changes * amacore agent integration * update liveness probe * handle non-existent file * refactor code * fix bugs in mdsd install * add poll to check amaca port up and running * fix bug * configure amaca configport * try released mdsd version 1.30.3 * fix bug in logs and events profile * test latest version of mdsd in GIG mode for both arm and x64 * try with build 50k eps changes * update templates for high log scale mode * remove libc.so copying * revert logrotate conf for amaca log * update mdsd version which has crash fix * add proxy support for amacore agent * update mdsd build with amaca gig la changes * update mdsd build with gig la fixes * update windows ama build * mdsd version with 25k buffer size in mdsd * update mdsd build * add telemetry and configmap option * fix bugs * windows ama build with resource id bug fix * update mdsd version with qos fixes * update to use working templates * add frequency to control amaca log * mdsd build with qos updates * trivy ignore update * log amaca agent version * improve comments * add default fluent-bit config for high log scale * add threding on tail plugin when high log scale enabled * fix bugs * fix bug * fix bugs * some improvements * improve comments * improve code * update trivyignore * fix bug * update trivyignore * pick GIGLA stream from config when highlogscale enabled * fix bug * template updates for high log scale mode * fix bug * clean up * set envvar for ishighlogscale * set envvar for ishighlogscale * fix bug * add log message to troubleshoot duplicate logs * add log message to troubleshoot duplicate logs * handle ama bug until fixed * add storage total limit size * rename for better reading * fix pr feedback * fix pr feedback * fix pr feedback * mdsd version update * fix proxy bug * fix proxy bug * update trivy ignore * clean up the code * refactor code * increase storage limit size to 2GB * increase storage limit size to 10GB * official mdsd and windows ama versions * code cleanup * code cleanup * mdsd version annotation update * fix pr feedback * fix pr feedback * fix pr feedback * fix pr feedback --------- Co-authored-by: Ganga Mahesh Siddem <gangams@microsoft.com> Co-authored-by: Amol Agrawal <pfrcks@gmail.com> Co-authored-by: Amol Agrawal <amagraw@microsoft.com> Co-authored-by: Janvi Jatakia (from Dev Box) <jajataki@microsoft.com>

* mdsd version 50k changes * amacore agent integration * update liveness probe * handle non-existent file * refactor code * fix bugs in mdsd install * add poll to check amaca port up and running * fix bug * configure amaca configport * try released mdsd version 1.30.3 * fix bug in logs and events profile * test latest version of mdsd in GIG mode for both arm and x64 * try with build 50k eps changes * update templates for high log scale mode * remove libc.so copying * revert logrotate conf for amaca log * update mdsd version which has crash fix * add proxy support for amacore agent * update mdsd build with amaca gig la changes * update mdsd build with gig la fixes * update windows ama build * mdsd version with 25k buffer size in mdsd * update mdsd build * add telemetry and configmap option * fix bugs * windows ama build with resource id bug fix * update mdsd version with qos fixes * update to use working templates * add frequency to control amaca log * mdsd build with qos updates * trivy ignore update * log amaca agent version * improve comments * add default fluent-bit config for high log scale * add threding on tail plugin when high log scale enabled * fix bugs * fix bug * fix bugs * some improvements * improve comments * improve code * update trivyignore * fix bug * update trivyignore * pick GIGLA stream from config when highlogscale enabled * fix bug * template updates for high log scale mode * fix bug * clean up * set envvar for ishighlogscale * set envvar for ishighlogscale * fix bug * add log message to troubleshoot duplicate logs * add log message to troubleshoot duplicate logs * handle ama bug until fixed * add storage total limit size * rename for better reading * fix pr feedback * fix pr feedback * fix pr feedback * mdsd version update * fix proxy bug * fix proxy bug * update trivy ignore * clean up the code * refactor code * increase storage limit size to 2GB * increase storage limit size to 10GB * official mdsd and windows ama versions * code cleanup * code cleanup * mdsd version annotation update * fix pr feedback * fix pr feedback * fix pr feedback * fix pr feedback

This reverts commit 69f3e8e.

ganga1980 added 30 commits March 21, 2024 21:11

mdsd version 50k changes

d47a8dd

Merge branch 'ci_prod' into gangams/logs-50k-eps-per-node

ca13e65

amacore agent integration

788517c

update liveness probe

78c41ea

handle non-existent file

38e36ab

refactor code

deaac81

fix bugs in mdsd install

117e482

add poll to check amaca port up and running

d617733

fix bug

ce75763

configure amaca configport

ad332a1

try released mdsd version 1.30.3

2065347

fix bug in logs and events profile

6414d76

test latest version of mdsd in GIG mode for both arm and x64

2f15da9

try with build 50k eps changes

ffe69cd

Merge branch 'ci_prod' into gangams/logs-50k-eps-per-node

64962de

Merge branch 'ci_prod' into gangams/logs-50k-eps-per-node

5e95d37

update templates for high log scale mode

47993d4

remove libc.so copying

623368d

Merge branch 'ci_prod' into gangams/logs-50k-eps-per-node

fd4fa2a

revert logrotate conf for amaca log

d60461b

update mdsd version which has crash fix

6388f2d

add proxy support for amacore agent

a3290d0

update mdsd build with amaca gig la changes

c724603

Merge branch 'ci_prod' into gangams/logs-50k-eps-per-node

c08de4f

update mdsd build with gig la fixes

feaf3e8

update windows ama build

132906e

mdsd version with 25k buffer size in mdsd

85fea6f

update mdsd build

6199823

add telemetry and configmap option

d822496

fix bugs

8f94ef1

ganga1980 added 5 commits May 4, 2024 19:27

update trivy ignore

bdd1476

clean up the code

6640315

refactor code

6656b4e

increase storage limit size to 2GB

2ffc627

increase storage limit size to 10GB

d2a316d

ganga1980 added 6 commits May 6, 2024 22:48

merge latest prod changes

5fdd0e4

official mdsd and windows ama versions

725f060

code cleanup

81e4ec3

code cleanup

e9a0888

mdsd version annotation update

de0ebbd

merge latest prod

4fbcb39

pfrcks reviewed May 9, 2024

View reviewed changes

build/linux/installer/conf/fluent-bit-common.conf Outdated Show resolved Hide resolved

pfrcks reviewed May 9, 2024

View reviewed changes

build/linux/installer/scripts/dcr-config-parser.rb Show resolved Hide resolved

pfrcks reviewed May 9, 2024

View reviewed changes

build/common/installer/scripts/fluent-bit-conf-customizer.rb Outdated Show resolved Hide resolved

ganga1980 added 3 commits May 9, 2024 10:34

fix pr feedback

01bcea5

fix pr feedback

758141d

fix pr feedback

a8b3986

jatakiajanvi12 reviewed May 9, 2024

View reviewed changes

fix pr feedback

062c9b4

pfrcks reviewed May 9, 2024

View reviewed changes

kubernetes/linux/main.sh Show resolved Hide resolved

ganga1980 enabled auto-merge (squash) May 9, 2024 23:37

pfrcks approved these changes May 9, 2024

View reviewed changes

ganga1980 merged commit 84fb709 into ci_prod May 9, 2024
15 checks passed

jatakiajanvi12 added a commit that referenced this pull request May 10, 2024

Revert "Gangams/logs 50k eps per node (#1235)"

8756af2

This reverts commit 69f3e8e.

jatakiajanvi12 added a commit that referenced this pull request May 10, 2024

Revert "Gangams/logs 50k eps per node (#1235)"

7067e87

This reverts commit 69f3e8e.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Gangams/logs 50k eps per node #1235

Gangams/logs 50k eps per node #1235

ganga1980 commented Apr 29, 2024 •

edited

ganga1980 commented May 7, 2024

ganga1980 commented May 7, 2024

ganga1980 commented May 9, 2024 •

edited

Gangams/logs 50k eps per node #1235

Gangams/logs 50k eps per node #1235

Conversation

ganga1980 commented Apr 29, 2024 • edited

ganga1980 commented May 7, 2024

ganga1980 commented May 7, 2024

ganga1980 commented May 9, 2024 • edited

ganga1980 commented Apr 29, 2024 •

edited

ganga1980 commented May 9, 2024 •

edited