Embed backend docker command #200

jt-dd · 2024-06-06T18:05:13Z

Adding backend docker command to remove kubehound.sh bash script.

./bin/build/kubehound backend --help
Handle the kubehound stack - docker compose based stack for kubehound services (mongodb, graphdb and UI)

Usage:
  kubehound backend [command]

Available Commands:
  down        Tear down the kubehound stack
  reset       Restart the kubehound stack
  up          Spawn the kubehound stack
  wipe        Wipe the persisted backend data

Flags:
  -f, --file strings   Compose configuration files
  -h, --help           help for backend

Global Flags:
      --debug   Enable debug logs

Use "kubehound backend [command] --help" for more information about a command.

Also, the backend gets automatically spawned when using the basic command ./kubehound.

Also, added an hidden command for all the spawning of the backend for the dev env:

./bin/build/kubehound dev --help
[devOnly] Spawn the kubehound dev stack for the system-tests (build from dockerfile)

Usage:
  kubehound dev [flags]
  kubehound dev [command]

Available Commands:
  system-tests [devOnly] Spawn the kubehound testing stack for the system-tests

Flags:
      --down   Tearing down the kubehound dev stack and deleting the data associated with it
  -h, --help   help for dev
      --ui     Tearing down the kubehound dev stack and deleting the data associated with it

Global Flags:
      --debug   Enable debug logs

Use "kubehound dev [command] --help" for more information about a command.

Added also a way to dump the local version : kubehound version

edznux-dd · 2024-06-07T08:12:37Z

deployments/kubehound/embed.go

+const (
+	DefaultComposePath = "docker-compose.embed.yaml"
+)
+
+//go:embed docker-compose.embed.yaml
+var F embed.FS


Just a note, as I'm not sure exactly how you want to handle this but:
you can use var F string directly, and avoid the need to readfile / do the error checks / convert to byte slice/string.

This also removes the need for the DefaultComposePath.

(if you want to provide another file, dynamically, you'll also need to change the FS, but that can be done on the caller or here, depends)

edznux-dd

Couple question / notes:

You haven't deleted the kubehound-ingestor directory, is that intended?
If an env is already running (or something is already listening on a port), it's spitting out the "help", which I don't think is helpful. IMO there should be a prompt asking to recreate the env. (We should probably ask before doing the action, in case there's some data that the user don't want to loose)
Could you add a print of the context that kubehound without any argument will run in? With enviroment with multiple context, this will get confusing if we are not very explicit about the env that we are pulling the data from by default.

On my machine, the janusgraph is failling on:

 ✘ Container kubehound-release-kubegraph-1  Error                                                                                                                                                                                                       66.4s 
 ✔ Container kubehound-release-ui-1         Started                                                                                                                                                                                                      0.5s 
 ✔ Container kubehound-release-mongodb-1    Created                                                                                                                                                                                                      0.0s 
Error: dependency failed to start: container kubehound-release-kubegraph-1 is unhealthy

waiting for JanusGraph Server...
/etc/opt/janusgraph/janusgraph-server.yaml will be used to start JanusGraph Server in foreground
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/log4j-slf4j-impl-2.20.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/janusgraph/lib/logback-classic-1.2.11.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
17:08:46 INFO  org.janusgraph.graphdb.server.JanusGraphServer.printHeader -                                                                       
   mmm                                mmm                       #     
     #   mmm   m mm   m   m   mmm   m"   "  m mm   mmm   mmmm   # mm  
     #  "   #  #"  #  #   #  #   "  #   mm  #"  " "   #  #" "#  #"  # 
     #  m"""#  #   #  #   #   """m  #    #  #     m"""#  #   #  #   # 
 "mmm"  "mm"#  #   #  "mm"#  "mmm"   "mmm"  #     "mm"#  ##m#"  #   # 
                                                         #            
                                                         "            

17:08:46 INFO  com.jcabi.log.Logger.infoForced - 0 attributes loaded from 330 stream(s) in 39ms, 104 saved, 5179 ignored: []
17:08:46 INFO  org.janusgraph.graphdb.server.JanusGraphServer.printHeader - JanusGraph Version: 1.0.0-rc2
17:08:46 INFO  org.janusgraph.graphdb.server.JanusGraphServer.printHeader - TinkerPop Version: 3.6.2
17:08:46 INFO  org.janusgraph.graphdb.server.JanusGraphServer.start - Configuring JanusGraph Server from /etc/opt/janusgraph/janusgraph-server.yaml
17:08:46 INFO  org.apache.tinkerpop.gremlin.server.util.MetricManager.addJmxReporter - Configured Metrics JmxReporter configured with domain= and agentId=
17:08:46 INFO  org.apache.commons.beanutils.FluentPropertyBeanIntrospector.introspect - Error when creating PropertyDescriptor for public final void org.apache.commons.configuration2.AbstractConfiguration.setProperty(java.lang.String,java.lang.Object)! Ignoring this property.
17:08:47 INFO  org.janusgraph.graphdb.idmanagement.UniqueInstanceIdRetriever.getOrGenerateUniqueInstanceId - Generated unique-instance-id=ac1113031-b9e1c66ac9961
17:08:47 INFO  org.janusgraph.diskstorage.Backend.getIndexes - Configuring index [search]
17:08:47 INFO  org.janusgraph.diskstorage.configuration.ExecutorServiceBuilder.buildFixedExecutorService - Initiated fixed thread pool of size 40
17:08:47 INFO  org.janusgraph.graphdb.database.StandardJanusGraph.<init> - Gremlin script evaluation is disabled
17:08:47 ERROR org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init> - Could not invoke constructor on class org.janusgraph.graphdb.management.JanusGraphManager (defined by the 'graphManager' setting) with one argument of class Settings
17:08:47 ERROR org.janusgraph.graphdb.server.JanusGraphServer.lambda$main$0 - JanusGraph Server was unable to start and will now begin shutdown
java.lang.RuntimeException: java.lang.reflect.InvocationTargetException
        at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.<init>(ServerGremlinExecutor.java:97) ~[gremlin-server-3.6.2.jar:3.6.2]

I have 5GB disk available, could do some cleanup but that seems large?
If not, we should say so somewhere, 5GB in VM and stuff it's a bunch

Will do another pass tomorow but LGTM overall :)

edznux-dd · 2024-06-10T16:24:57Z

pkg/backend/util.go

@@ -0,0 +1,23 @@
+package backend
+
+func mergeMaps(currentMap map[interface{}]interface{}, newMap map[interface{}]interface{}) map[interface{}]interface{} {


Could you write a test for this? :D (maybe create a card for later).
It seems simple but the amount of interface{}... :D

Btw, you can use any instead, it'll be a tad cleaner I guess.

cmd/kubehound/dev.go

edznux-dd · 2024-06-10T16:27:56Z

cmd/kubehound/dev.go

+	envCmd = &cobra.Command{
+		Use:    "dev",
+		Hidden: true,


Probably add a line in the readme / how to contribute to document this if we hide that command. (I will forget this in 3month)

adding in the Jira card to track it for later (when new release is official).

edznux-dd · 2024-06-10T16:33:20Z

cmd/kubehound/dev.go

+			var err error
+			Backend, err = docker.NewBackend(cobraCmd.Context(), composePath)
+
+			return err


Backend is a global var right? does it need to be exported? (it's also set in the runEnv function), which i'm not sure on why aren't using as well?

It looks like it's doing that twice: one in the pre run and once in the run.

(I feel like the Backend should be a singleton in another package, rather than in the cmd/kubehound, probably in the pkg/backend directly tbh)

added the singleton as we discussed

cmd/kubehound/version.go

edznux-dd · 2024-06-10T16:40:17Z

deployments/kubehound/embed.go

@@ -0,0 +1,8 @@
+package embedconfigdocker


Nit: I'm always in favor of using the folder as the package name and we can always rename the import.
I find it easier for adding new files and all.

For this "use case" where it's likely to be the only file, i'm fine with it but 🤷

edznux-dd · 2024-06-10T16:43:54Z

pkg/backend/containers.go

+type Backend struct {
+	project        *types.Project
+	composeService api.Service
+	dockerCli      *command.DockerCli
+}
+
+func NewBackend(ctx context.Context, composeFilePaths []string) (*Backend, error) {


I think this package would benefit from using a singleton pattern mentionned in the cmd package earlier.
There's only one backend that should be running at the same time right, and it'll remove the global var there.

pkg/backend/containers.go

pkg/backend/project.go

pkg/backend/containers.go

edznux-dd

LFG! ✔️ 🚀

jt-dd added 2 commits June 6, 2024 20:01

embed backend commands into kubehound binary

2baa774

adding force pulling

6e7acb0

jt-dd temporarily deployed to devenv June 6, 2024 18:05 — with GitHub Actions Inactive

fix

92ec233

jt-dd temporarily deployed to devenv June 6, 2024 18:15 — with GitHub Actions Inactive

edznux-dd reviewed Jun 7, 2024

View reviewed changes

adding filepath input

f02b082

jt-dd temporarily deployed to devenv June 7, 2024 13:44 — with GitHub Actions Inactive

jt-dd added 4 commits June 7, 2024 22:36

porting all makefile command to binary

f36a2a6

fixing kubehound config file input

5b2951c

adding support for datadog local env

84eb199

fix datadog config

80ec69c

jt-dd temporarily deployed to devenv June 7, 2024 21:09 — with GitHub Actions Inactive

fix linter

62c54ef

jt-dd temporarily deployed to devenv June 7, 2024 21:17 — with GitHub Actions Inactive

jt-dd added 2 commits June 8, 2024 14:04

hidding dev command

47cea08

adding version command

804d0c8

jt-dd temporarily deployed to devenv June 8, 2024 12:05 — with GitHub Actions Inactive

fix docker-compose

8170bcb

jt-dd temporarily deployed to devenv June 8, 2024 13:10 — with GitHub Actions Inactive

remove poc file

04cba2d

jt-dd temporarily deployed to devenv June 8, 2024 13:17 — with GitHub Actions Inactive

own command for version

38cdabb

jt-dd temporarily deployed to devenv June 9, 2024 10:28 — with GitHub Actions Inactive

jt-dd added 2 commits June 10, 2024 15:09

fix conflict between commands

81a8da7

fixing Dockerfile ingestor image

8271e8c

jt-dd temporarily deployed to devenv June 10, 2024 13:23 — with GitHub Actions Inactive

jt-dd added 3 commits June 10, 2024 17:46

new buildx stack

723a97b

update go.mod

a63ed32

adding arch/os to version command

5d7a981

jt-dd temporarily deployed to devenv June 10, 2024 16:11 — with GitHub Actions Inactive

jt-dd added 2 commits June 10, 2024 18:29

fix system-tests

214d1c5

fix UI flag dev command

946110b

jt-dd temporarily deployed to devenv June 10, 2024 16:38 — with GitHub Actions Inactive

fix system-test

633a6dd

jt-dd temporarily deployed to devenv June 10, 2024 16:45 — with GitHub Actions Inactive

edznux-dd reviewed Jun 10, 2024

View reviewed changes

fix system-tests

0fc2008

jt-dd temporarily deployed to devenv June 10, 2024 17:27 — with GitHub Actions Inactive

PR comment

b3f8516

jt-dd temporarily deployed to devenv June 10, 2024 17:29 — with GitHub Actions Inactive

deleting kubehoud-ingestor

d932f9d

jt-dd temporarily deployed to devenv June 10, 2024 17:32 — with GitHub Actions Inactive

jt-dd added 2 commits June 10, 2024 19:44

pR comment

2a590b9

PR comment (singleton)

2c6751f

jt-dd temporarily deployed to devenv June 10, 2024 17:58 — with GitHub Actions Inactive

jt-dd added 2 commits June 10, 2024 19:58

PR comment (singleton)

8c20e8a

private method for backend object

726e03e

jt-dd temporarily deployed to devenv June 10, 2024 18:00 — with GitHub Actions Inactive

fix linter

2839bf4

jt-dd temporarily deployed to devenv June 11, 2024 07:47 — with GitHub Actions Inactive

jt-dd added 2 commits June 11, 2024 09:54

ignore linter

0a2f470

fixing noise aournd errors

dc4370e

jt-dd temporarily deployed to devenv June 11, 2024 08:13 — with GitHub Actions Inactive

jt-dd marked this pull request as ready for review June 11, 2024 08:17

jt-dd requested a review from a team as a code owner June 11, 2024 08:17

edznux-dd approved these changes Jun 11, 2024

View reviewed changes

jt-dd merged commit 8b7951b into main Jun 11, 2024
7 checks passed

jt-dd deleted the jt-dd/backend-commands-embedded branch June 11, 2024 08:43

edznux-dd mentioned this pull request Jun 13, 2024

Added basic checks to ensure the correct version of Docker Compose is installed. #155

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Embed backend docker command #200

Embed backend docker command #200

jt-dd commented Jun 6, 2024 •

edited

Loading

edznux-dd Jun 7, 2024

edznux-dd left a comment •

edited

Loading

edznux-dd Jun 10, 2024

edznux-dd Jun 10, 2024

jt-dd Jun 10, 2024

edznux-dd Jun 10, 2024

jt-dd Jun 10, 2024

edznux-dd Jun 10, 2024

edznux-dd Jun 10, 2024

jt-dd Jun 11, 2024

edznux-dd left a comment •

edited

Loading

		@@ -0,0 +1,23 @@
		package backend

		func mergeMaps(currentMap map[interface{}]interface{}, newMap map[interface{}]interface{}) map[interface{}]interface{} {

Embed backend docker command #200

Embed backend docker command #200

Conversation

jt-dd commented Jun 6, 2024 • edited Loading

Choose a reason for hiding this comment

edznux-dd left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

edznux-dd left a comment • edited Loading

Choose a reason for hiding this comment

jt-dd commented Jun 6, 2024 •

edited

Loading

edznux-dd left a comment •

edited

Loading

edznux-dd left a comment •

edited

Loading