Document effective ways to run Cartography in Kubernetes

[achantavy]

Description:

Describe your idea. Please be detailed. If a feature request, please
describe the desired behavior, what scenario it enables, and how it
would be used.

Kubernetes has officially taken over the world and we should document ways to run Cartography in a "suitcase"-based, binpacked, container world.

This is a follow-up to #159. Configuring Neo4j in Docker is out of the scope of this issue because there are lots of great existing docs on how to do that, but we could provide a document that describes cartography-specific concerns such as

• How to split our very large sync into smaller, more flexible K8s cronjobs using custom scripts and choosing to sync smaller parts of a sync. (the interesting thing here is that you end up getting some degree of parallelism for "free"!)
• Gotchas with K8s load balancers/ingress and timeouts

We should also link to @marco-lancini's great blog posts on making this happen :)

• https://www.marcolancini.it/2020/blog-mapping-moving-clouds-with-cartography/
• https://www.marcolancini.it/2020/blog-tracking-moving-clouds-with-cartography/

[marco-lancini]

Awesome!
I'm about to work on releasing a vanilla deployment of Cartography on Kubernetes as part of my Kubernetes Lab: https://github.com/marco-lancini/k8s-lab-plz

It should be ready in a couple of weeks

[krisek]

I have already a little Ansible playbook and a Dockerfile doing the job

FROM python:3.9-alpine as base

RUN apk add --update-cache git gcc musl-dev libffi-dev openssl-dev make

FROM base as carto

RUN pip install https://github.com/krisek/cartography/archive/a3100d457f642c55bc9317a952bf8daca34f48c7.zip

CMD cartography

- hosts: "{{ variable_host | default('') }}" 
#ansible-galaxy collection install --force-with-deps community.aws 
  connection: local
  gather_facts: no
  become: no
  vars:
    awscli_saml_python_path:  /home/kris/projects/awscli-saml/venv/bin/python3
    awscli_saml_path: /home/kris/projects/awscli-saml/aws_saml_cli.py
    vipaccess_path: /home/kris/Nextcloud/projects/tsl/venv/bin/vipaccess 
    vipaccess_cred_path: /media/kris/CM/vipaccess
    cartography_account: ''
    neo4j_uri: 'bolt://neo4j:7687'
  tasks:
  - name: show accounts
    debug:
      msg: "{{ inventory_hostname }}"

  - name: get tsl token
    command: "{{ vipaccess_path }} show -f {{ vipaccess_cred_path}}"
    register: tsl_token
    tags: ['never', 'login']

  - name: login
    command: "{{ awscli_saml_python_path }}  {{ awscli_saml_path }} -p saml_{{ inventory_hostname }} -t {{ tsl_token.stdout}} -a '{{ inventory_hostname }}'"
    tags: ['never','login']

  - name: get session token for boto
    set_fact: 
      session_token: "{{ lookup('ini', 'aws_session_token section=saml_{{ inventory_hostname }} file=/home/kris/.aws/credentials') }}"

  - name: delete policies of readonly user
    community.aws.iam_user:
      name: cartography_crawler
      managed_policies:
        - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
        - arn:aws:iam::aws:policy/SecurityAudit
      state: absent
      profile: "saml_{{ inventory_hostname }}"


  - name: delete user with api key
    community.aws.iam:
      iam_type: user
      name: cartography_crawler
      state: absent
      profile: "saml_{{ inventory_hostname }}"
      security_token: "{{ session_token }}"


  - name: create user with api key
    community.aws.iam:
      iam_type: user
      name: cartography_crawler
      state: present
      access_key_state: create
      profile: "saml_{{ inventory_hostname }}"
      security_token: "{{ session_token }}"
    register: user_result_1

  - name: create a readonly user
    community.aws.iam_user:
      name: cartography_crawler
      managed_policies:
        - arn:aws:iam::aws:policy/job-function/ViewOnlyAccess
        - arn:aws:iam::aws:policy/SecurityAudit
      state: present
      profile: "saml_{{ inventory_hostname }}"
    register: user_result_2

  - debug:
      msg: "{{ user_result_1 }}"
    tags: ['never']

  - name: aws tag user
    command: "aws iam tag-user --profile saml_{{ inventory_hostname }} --user-name cartography_crawler  --tags Key=Environment,Value=PROD Key=Name,Value=cartography_crawler"


  - name: create cronjob
    community.kubernetes.k8s:
      state: present
      template: 
        path: cronjob.yaml
      context: "automation"

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: cartography-{{ inventory_hostname }}
  namespace: asset-management
spec:
  schedule: "{{ 60 | random(step=5) }} 3,13 * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cartography-{{ inventory_hostname }}
            image: cartography:latest
            env:
            - name: AWS_DEFAULT_REGION
              value: eu-central-1
            - name: AWS_ACCESS_KEY_ID
              value: {{ user_result_1.user_meta.access_keys.0.access_key_id }}
            - name: AWS_SECRET_ACCESS_KEY
              value: {{  user_result_1.user_meta.access_keys.0.secret_access_key  }}
            command: ["cartography"]
            args: ["--neo4j-uri", "{{ neo4j_uri }}"]
          restartPolicy: OnFailure⏎ 

It's rough at the edges for the time being, but you can get the idea. It creates a IAM account for each of your accounts, and installs a k8s CronJob for each of them.

Plan is to write a nice guide to medium.com as soon as time permits.

[marco-lancini]

Awesome!
I'm about to work on releasing a vanilla deployment of Cartography on Kubernetes as part of my Kubernetes Lab: https://github.com/marco-lancini/k8s-lab-plz

It should be ready in a couple of weeks

And there it is: https://github.com/marco-lancini/k8s-lab-plz/tree/master/components/cartography
Cartography+Neo4j running on Kubernetes, using Vault as secrets management solution.

Companion blog post: https://www.marcolancini.it/2021/blog-cartography-on-kubernetes/

@achantavy : keen to hear your thoughts on this

[achantavy]

@marco-lancini - you wanna add this to a section in docs/ and close out this issue? :)

[marco-lancini]

Could do, what about https://github.com/lyft/cartography/blob/master/docs/setup/install.md?

[achantavy]

@marco-lancini Could add a file in https://github.com/lyft/cartography/tree/master/docs/containers and then link to it in https://github.com/lyft/cartography/blob/master/docs/setup/install.md. Just need to make sure the install flow is still smooth and understandable after our edits.

@krisek For some reason I only saw your comment right now and missed it even though you posted it 25 days ago - sorry! Definitely would be happy to include your doc also in containers/.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new change or reopen this one.