Onboarding Kafka benchmark workload [Windows] #259

mohitgole · 2024-02-02T10:54:52Z

Added kafka workload support for windows x64 and arm64.

Added server/client executors.
Added producer/consumer parsers.
Added unit tests.
Added functional tests.
Added documentation.

Note: Support for linux OS will be raised post current windows PR

…o users/mohitgole/kafka-workload-onboard

…th latest changes in JDKPackageDependencyInstallation

… changes

…o users/mohitgole/kafka-workload-onboard

yangpanMS · 2024-04-18T21:48:50Z

src/VirtualClient/VirtualClient.Main/profiles/PERF-KAFKA.json

+      "Parameters": {
+        "Scenario": "DownloadKafkaPackage",
+        "BlobContainer": "packages",
+        "BlobName": "kafka_2.13-3.6.1.zip",


Can you document how this was packaged? And is it a 2.13 version or 3.6.1 version?

Yeah, so here 2.13 is scala version and 3.6.1 is kafka version.

yangpanMS · 2024-04-18T21:50:09Z

src/VirtualClient/VirtualClient.Main/VirtualClient.Main.csproj

-        <ItemGroup>
-            <ScriptFiles Include="$(RepoPath)\out\scripts\**\*.*" />
-        </ItemGroup>
+	<Target Name="CopyInScriptsToPublishDirectory" AfterTargets="Publish">


Please confirm your tab is 4 spaces.

yangpanMS · 2024-04-18T21:51:14Z

src/VirtualClient/VirtualClient.Main/CommandBase.cs

@@ -292,6 +292,11 @@ protected virtual IServiceCollection InitializeDependencies(string[] args)
                this.LoggingLevel = LogLevel.Trace;
            }

+            if (this.Debug)


We already have this in, why do you need to add it?

Yeah this looks like something came from merge conflict. Will remove it.

yangpanMS · 2024-04-18T21:52:44Z

src/VirtualClient/VirtualClient.Core/VirtualClientComponentExtensions.cs

@@ -108,7 +113,9 @@ public static class VirtualClientComponentExtensions
            CancellationToken cancellationToken,
            bool runElevated = false,
            string username = null,
-            Action<IProcessProxy> beforeExecution = null)
+            Action<IProcessProxy> beforeExecution = null,
+            TimeSpan? timeout = null,


Please let me know why this change was needed.

As discussed, creating similar method in KafkaExecutor and removing change in VirtualClientComponentExtensions.

yangpanMS · 2024-04-18T21:54:35Z

src/VirtualClient/VirtualClient.Actions/Kafka/KafkaExecutor.cs

+    /// </summary>
+    [UnixCompatible]
+    [WindowsCompatible]
+    public class KafkaExecutor : VirtualClientComponent


make it abstract if not intended for direct use and call it like KafkaBaseExecutor

yangpanMS · 2024-04-18T21:55:31Z

src/VirtualClient/VirtualClient.Actions.UnitTests/Kafka/KafkaProducerMetricsParserTests.cs

+        public void KafkaProducerMetricsParserThrowIfInvalidOutputFormat()
+        {
+            string workingDirectory = Path.GetDirectoryName(Assembly.GetExecutingAssembly().Location);
+            string outputPath = Path.Combine(workingDirectory, @"Examples\Kafka\KafkaProducerResultExample.txt");


Use platform agnostic Path.Combine so that this works in linux unit test

Yes made the change.

…o users/mohitgole/kafka-workload-onboard

yangpanMS · 2024-04-24T18:48:29Z

website/docs/overview/overview.md

@@ -91,6 +91,7 @@ for using them.
 | [Stressapptest](https://microsoft.github.io/VirtualClient/docs/workloads/stressapptest) | Fault Tolerance | linux-x64, linux-arm64 | [Apache-2.0](https://github.com/stressapptest/stressapptest/blob/master/NOTICE)  |
 | [Stress-ng](https://microsoft.github.io/VirtualClient/docs/workloads/stress-ng) | Fault Tolerance | linux-x64, linux-arm64 | [GPL-2.0](https://github.com/ColinIanKing/stress-ng/blob/master/COPYING)  |
 | [SuperBench](https://microsoft.github.io/VirtualClient/docs/workloads/superbenchmark) | Machine Learning | linux-x64 | [MIT](https://github.com/microsoft/superbenchmark/blob/main/LICENSE)  |
+| [Kafka](../workloads/kafka/kafka.md) | Streaming data | win-x64, win-arm64 | [Apache License 2.0](https://github.com/apache/kafka/blob/trunk/LICENSE)  |


Just Apache-2.0 to align with the others

yangpanMS · 2024-04-24T18:49:44Z

website/docs/workloads/kafka/kafka.md

+call :concat "%BASE_DIR%\libs\*;"
+```
+
+For reference [The input line is too long](https://github.com/kafka-dev/kafka/issues/61)


Thank you for documenting this

yangpanMS · 2024-04-24T18:50:58Z

website/docs/workloads/kafka/kafka-profiles.md

+
+``` bash
+# Single System (environment layout not required)
+./VirtualClient --profile=PERF-KAFKA.json --system=Juno --timeout=1440


Please remove all Juno references. If you find it elsewhere please remove too.

yangpanMS · 2024-04-24T18:51:23Z

src/VirtualClient/VirtualClient.Actions/Kafka/KafkaServerExecutor.cs

@@ -0,0 +1,498 @@
+using System;


Add mit header please

yangpanMS · 2024-04-24T18:52:15Z

src/VirtualClient/VirtualClient.Actions/Kafka/KafkaServerExecutor.cs

+        /// <summary>
+        /// Port on which server runs.
+        /// </summary>
+        public int Port


For workloads like these we can have a default port so profile can be shorter.

yangpanMS · 2024-04-24T18:55:12Z

src/VirtualClient/VirtualClient.Actions/Kafka/KafkaServerExecutor.cs

+
+        private async Task ConfigurePropertiesFileAsync(EventContext telemetryContext, CancellationToken cancellationToken)
+        {
+            string logDir = "/tmp";


Does this PR have Linux support or not? Just to make sure we are coding for both OS in the future

Current PR targets support for windows. Linux will be tested and raised as separate PR.

yangpanMS · 2024-04-24T21:38:23Z

src/VirtualClient/VirtualClient.Actions/Kafka/KafkaExecutor.cs

+            this.KafkaPackagePath = kafkaPackage.Path;
+
+            switch (this.Platform)
+            {


I don't believe you would need this. You don't need cmd to act as a process exe

yangpanMS · 2024-04-24T21:39:33Z

src/VirtualClient/VirtualClient.Actions/Kafka/KafkaClientExecutor.cs

+using VirtualClient.Contracts;
+using VirtualClient.Contracts.Metadata;
+
+namespace VirtualClient.Actions.Kafka


For namespace we can just to VirtualClient.Actions. Apply thoughout

yangpanMS

Would you be ok if you move this PR to the ADO so that we can vet it in QoS_VC internally first?

mohitgole · 2024-04-26T14:05:29Z

Would you be ok if you move this PR to the ADO so that we can vet it in QoS_VC internally first?

Created another ADO PR.

mohitgole added 29 commits October 27, 2023 07:03

[Kafka Onboard] - initial commit

e02ad46

Working server client kafka initial commit

6f3c248

Added producer and consumer parsers

b779f53

Changes such that producer can produce only once

0fd3a68

Merge branch 'main' of https://github.com/mohitgole/VirtualClient int…

d85a70f

…o users/mohitgole/kafka-workload-onboard

Moved function to base class

96fb679

Fixes to parser. Added timeout param to VirtualClientComponentExtensions

b85d008

Make partitions based on serverInstances

b9e40cc

[Fixed] - Working kafka workload in windows for single and multinode

74fca72

Reverted creation of ProcessCommandAsync function

ab35dc5

initial single working unit tests for server

5df02ec

Unit tests for kafka server, client and parsers

3d6d56d

Merge branch 'main' of https://github.com/mohitgole/VirtualClient int…

2425f47

…o users/mohitgole/kafka-workload-onboard

Fixed cast conversion

d64ea23

Commented KafkaServerState instances and usages

7dd4815

Added functional tests for kafka profile

cdcd397

Changed kafka version to 3.6.1

84c9d27

Merge branch 'main' of https://github.com/mohitgole/VirtualClient int…

c205a7b

…o users/mohitgole/kafka-workload-onboard

Minor change in kaka profile

4e2c9b4

Merge branch 'main' of https://github.com/mohitgole/VirtualClient int…

c420846

…o users/mohitgole/kafka-workload-onboard

Removed JavaDevelopmentKitInstallation from kafka profile to match wi…

bda181e

…th latest changes in JDKPackageDependencyInstallation

Removed unused Port param in kafka client

eb94cd2

Removed unused code

8ad06be

Removed time delay

5d838b0

Nits: unit tests

55bbf6b

Minor code improvements

f8c1968

Updated comment and port for server

4396024

Removed unused variable startTime

3049f22

Removed mentions of redis

9dfbd9b

mohitgole marked this pull request as ready for review February 2, 2024 10:59

mohitgole added 5 commits March 30, 2024 10:37

Merge conflict fix

21eb51b

Added ClientInstance, Partitions parameters. Fixed file configuration…

84c6f0a

… changes

Merge branch 'main' of https://github.com/mohitgole/VirtualClient int…

bd7f772

…o users/mohitgole/kafka-workload-onboard

Merge branch 'main' of https://github.com/mohitgole/VirtualClient int…

8675253

…o users/mohitgole/kafka-workload-onboard

Add parameter MaxClientInstances for validation

65adf5b

yangpanMS reviewed Apr 18, 2024

View reviewed changes

mohitgole added 6 commits April 23, 2024 16:20

Merge branch 'main' of https://github.com/mohitgole/VirtualClient int…

9da7451

…o users/mohitgole/kafka-workload-onboard

Addressed PR review comments

819c7ac

Removed extra line break

ebb206d

Fixed missed ;

795c2e1

Updated kafka.md file

4bf77b8

Added licence in document

9518e90

yangpanMS reviewed Apr 24, 2024

View reviewed changes

mohitgole added 2 commits April 25, 2024 16:59

Addressed review comments - 2

a0f03bc

Changed action name Setup to CreateTopic

9f7582f

yangpanMS closed this Jul 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Onboarding Kafka benchmark workload [Windows] #259

Onboarding Kafka benchmark workload [Windows] #259

mohitgole commented Feb 2, 2024 •

edited

Loading

yangpanMS Apr 18, 2024

mohitgole Apr 23, 2024

yangpanMS Apr 18, 2024

yangpanMS Apr 18, 2024

mohitgole Apr 23, 2024

yangpanMS Apr 18, 2024

mohitgole Apr 23, 2024

yangpanMS Apr 18, 2024

mohitgole Apr 23, 2024

yangpanMS Apr 18, 2024

mohitgole Apr 24, 2024

yangpanMS Apr 24, 2024

yangpanMS Apr 24, 2024

yangpanMS Apr 24, 2024

yangpanMS Apr 24, 2024

yangpanMS Apr 24, 2024

yangpanMS Apr 24, 2024

mohitgole Apr 25, 2024

yangpanMS Apr 24, 2024

yangpanMS Apr 24, 2024

yangpanMS left a comment

mohitgole commented Apr 26, 2024

Onboarding Kafka benchmark workload [Windows] #259

Onboarding Kafka benchmark workload [Windows] #259

Conversation

mohitgole commented Feb 2, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yangpanMS left a comment

Choose a reason for hiding this comment

mohitgole commented Apr 26, 2024

mohitgole commented Feb 2, 2024 •

edited

Loading