Demo Setup

[albrechtflo-hg]

Having trouble with the Demo Setup? Do you have tips for others? Let us know, so everyone can learn!

[ksilin]

Hi @albrechtflo-hg. The demo is running fine. I am missing a lot of information about the tool to get up and running and use it appropriately. To make it more concrete - the questions I had during the first couple of minutes:

• What are the typical workflows? Users and admins have several workflows
• Is there an API? Can I automate/script things, e.g. via Github Actions or a CI? How can I integrate it into an Infra-as-Code setup?
• What can be customized and how? Can one change the default partition counts? Can one change the CloudEvents prerequisite?
• Is there an audit function? So that all actions are recorded, ideally in a tamper-resistant format in a configurable sink.
• Can we migrate the state? Export, manipulate, reload. AsyncAPI maybe?
• Are there metrics to collect? JMX or Prometheus endpoint perhaps?
• Can Galapagos be integrated with AAD for users and groups?
• Is there a configurable authorization (see only resources with a certain prefix for example)?
  and many more.

I hope they have been answered elsewhere :)

Thank you!

[GunnarK]

Hi @ksilin!
Thank you for your good questions. Since these are quite a lot and some require a longer answer, please allow us some time to answer.
Meanwhile you might want to watch the presentation @albrechtflo-hg and I made at last years code.talk (german language). There Florian adresses the typical workflows.

[albrechtflo-hg]

Hi @ksilin , I will try to answer your points. Most of them do not have "the" single point where they are already documented now.

• Workflows: There are several, maybe too many to describe them here in detail (maybe that would really make a good new document). I will try to list the most common ones:
  • Browse for required events / data - using the list of topics and their description, find events or data which might be relevant for your use case
  • Subscribe to a topic - retrieve read access to desired topics. This may include an approval of the providing team, if the topic is marked as protected
  • Create a new topic (application internal or "API" / event topic)
  • Publish a new schema for an owned topic
  • Stage a topic to the next environment ("environment" is just a Kafka cluster and the associated meta-information in Galapagos)
  • Mark a topic as deprecated, and delete it after EOL or after all subscribers have canceled their subscription
  • retrieve short-lived "developer API keys" for personal use in external Kafka tools, e.g. to peek into topics in production
  • Admins really only have to approve requests of users for being "assigned" to an application, or remove their access later on. As soon as a user is assigned to an application, they can do most operations for "their" application without any additional approval (except for subscribing to protected topics).

There are additional admin workflows which may have to be executed on an irregular basis, e.g. protecting a topic after creation, creating API keys for external tools wanting to access Galapagos metadata in Kafka etc. For these tasks, there are so-called "admin jobs" in Galapagos which are executed via command line as one-time jobs.

• API / automation / IaC: Well, in principle, all Galapagos operations are accessible via REST API, as the frontend also only uses the REST API of the backend. But there really should be no need in usual operation. For example, a topic can be created and even staged to production in advance, long before an application is deployed to production. Same applies for most other operations, and, as these are all one-time operations being performed by a developer, there is no need for automation in our opinion (and, well, we use Galapagos for more than three years and with more than 10 DevOps teams and 60 users at our company now).
  An exception may be the full restore case, but we put this responsibility to the Kafka infrastructure (so, when one of our clusters is lost, the teams do not have the responsibility to restore their topics, which would be a good case for automation).

• Customization: See the application.properties file in src/main/resources, or the application-demo.yml file for potential configurations to customize Galapagos to your needs. There are some things, which cannot currently be customized, though (only via customization of the source code). The "CloudEvents prerequisite" is currently "just" a convention, nothing which is enforced on any level.

• Audit Function: Galapagos has a separate "audit" logger which can be routed to secure sinks using common logging configurations.

• State migration: There is a backup and import admin job for Galapagos metadata, which mainly serves backup purposes. Imported metadata is not applied to the cluster, so there is no (meaningful) workflow for manipulating metadata outside Galapagos (and that would miss the idea of the whole tool). AsyncAPI is, in this context, more "low-level" and would be directly used on cluster-level.

• Metrics: Currently, Galapagos does not export any meaningful metrics. As a pure developer tool with low criticality, we did not see any value in this. Do you have metrics you would be missing from such a tool? Let us hear your ideas!

• AAD Integration: Yes, using a lightweight Keycloak as an intermediate broker. We have exactly this setup for our Galapagos instances at our company. The user does not even notice the intermediate Keycloak, as the AAD IdP is the default IdP and automatically redirected to.

• Authorization: Complex topic with multiple levels. Having only access to required / approved resources in Kafka is one of the purposes of Galapagos, so yes, that is very fine-grained. Also, you can protect topics, so only "topic administrators" of applications with a valid subscriptions can view the contents of these topics (Galapagos offers a "preview" mode for topic data). Additionally, "app internal" topics of applications are not visible to topic administrators of other applications, not even in Galapagos. All "API" topics are, in their mere existence, visible to all Galapagos users in the Galapagos UI.

Please let us know additional questions or your impressions from using Galapagos. Yes, currently, many of these topics are lacking in-depth documentation. Creating a kind of "demo video" is on our backlog for a long time now, but for a commercial company, investing resources in promoting an open-source project is, unfortunately, low-priority most of the time.