-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automatically generate datanode.conf.example #19141
Open
todvora
wants to merge
11
commits into
master
Choose a base branch
from
feature/automatic-datanode.conf.example
base: master
Could not load branches
Branch not found: {{ refName }}
Could not load tags
Nothing to show
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 4 commits
Commits
Show all changes
11 commits
Select commit
Hold shift + click to select a range
db97490
Automatically generate datanode.conf.example + csv documentation of c…
todvora 228862a
code cleanup, tests
todvora a8656ca
Merge branch 'master' into feature/automatic-datanode.conf.example
todvora 1b959c0
Merge branch 'master' into feature/automatic-datanode.conf.example
todvora fe6ce97
Merge branch 'master' into feature/automatic-datanode.conf.example
todvora c031465
Merge branch 'master' into feature/automatic-datanode.conf.example
todvora 4bf1ff6
add spaces around = in generated config example
todvora ffed46e
adjust spaces in generated config files, add tests for ordering
todvora dfc5cda
Merge branch 'master' into feature/automatic-datanode.conf.example
todvora 06022c4
Merge branch 'master' into feature/automatic-datanode.conf.example
todvora 5a1c5d2
Added support for sections - both on class level and field level
todvora File filter
Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
138 changes: 138 additions & 0 deletions
138
data-node/src/main/java/org/graylog/datanode/docs/ConfigurationDocsGenerator.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,138 @@ | ||
| /* | ||
| * Copyright (C) 2020 Graylog, Inc. | ||
| * | ||
| * This program is free software: you can redistribute it and/or modify | ||
| * it under the terms of the Server Side Public License, version 1, | ||
| * as published by MongoDB, Inc. | ||
| * | ||
| * This program is distributed in the hope that it will be useful, | ||
| * but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| * Server Side Public License for more details. | ||
| * | ||
| * You should have received a copy of the Server Side Public License | ||
| * along with this program. If not, see | ||
| * <http://www.mongodb.com/licensing/server-side-public-license>. | ||
| */ | ||
| package org.graylog.datanode.docs; | ||
|
|
||
| import com.github.joschi.jadconfig.Parameter; | ||
| import com.github.joschi.jadconfig.ReflectionUtils; | ||
| import jakarta.annotation.Nonnull; | ||
| import org.apache.commons.lang3.ClassUtils; | ||
| import org.graylog.datanode.commands.Server; | ||
| import org.graylog.datanode.docs.printers.ConfigFileDocsPrinter; | ||
| import org.graylog.datanode.docs.printers.CsvDocsPrinter; | ||
| import org.graylog.datanode.docs.printers.DocsPrinter; | ||
| import org.graylog2.configuration.Documentation; | ||
|
|
||
| import java.io.FileWriter; | ||
| import java.io.IOException; | ||
| import java.lang.reflect.Field; | ||
| import java.nio.charset.StandardCharsets; | ||
| import java.util.Arrays; | ||
| import java.util.Comparator; | ||
| import java.util.List; | ||
| import java.util.Locale; | ||
| import java.util.Optional; | ||
| import java.util.function.Supplier; | ||
| import java.util.stream.Stream; | ||
|
|
||
| public class ConfigurationDocsGenerator { | ||
|
|
||
| /** | ||
| * This class is linked from the datanode pom.xml and generates conf.example and csv documentation. | ||
| */ | ||
| public static void main(String[] args) throws IOException { | ||
| final ConfigurationDocsGenerator generator = new ConfigurationDocsGenerator(); | ||
| generator.generateDocumentation(parseDocumentationFormat(args), ConfigurationDocsGenerator::getDatanodeConfigurationBeans); | ||
| } | ||
|
|
||
| @Nonnull | ||
| private static DocumentationFormat parseDocumentationFormat(String[] args) { | ||
| if (args.length != 2) { | ||
| throw new IllegalArgumentException("This command needs two arguments - a format and file path. For example" + "csv ${project.build.directory}/configuration-documentation.csv"); | ||
| } | ||
| final String format = args[0].toLowerCase(Locale.ROOT).trim(); | ||
| final String file = args[1]; | ||
| return new DocumentationFormat(format, file); | ||
| } | ||
|
|
||
| protected void generateDocumentation(DocumentationFormat format, Supplier<List<Object>> configurationBeans) throws IOException { | ||
| final List<ConfigurationEntry> configuration = detectConfigurationFields(configurationBeans); | ||
| try (final DocsPrinter writer = createWriter(format)) { | ||
| writer.writeHeader(); | ||
|
|
||
| for (ConfigurationEntry f : configuration) { | ||
| writer.writeField(f); | ||
| } | ||
| } | ||
| } | ||
|
|
||
| private DocsPrinter createWriter(DocumentationFormat format) throws IOException { | ||
| final FileWriter fileWriter = new FileWriter(format.outputFile(), StandardCharsets.UTF_8); | ||
| return switch (format.format()) { | ||
| case "csv" -> new CsvDocsPrinter(fileWriter); | ||
| case "conf" -> new ConfigFileDocsPrinter(fileWriter); | ||
| default -> throw new IllegalArgumentException("Unsupported format " + format.format()); | ||
| }; | ||
| } | ||
|
|
||
| /** | ||
| * Collects all configuration options from all available configuration beans. | ||
| */ | ||
| private List<ConfigurationEntry> detectConfigurationFields(Supplier<List<Object>> configurationBeans) { | ||
| return configurationBeans.get() | ||
| .stream() | ||
| .flatMap(ConfigurationDocsGenerator::beanToConfigEntries) | ||
| .sorted(Comparator.comparing(ConfigurationEntry::isPriority).reversed()) | ||
| .toList(); | ||
| } | ||
|
|
||
| @Nonnull | ||
| private static Stream<ConfigurationEntry> beanToConfigEntries(Object configurationBean) { | ||
| return Arrays.stream(configurationBean.getClass().getDeclaredFields()) | ||
| .filter(f -> f.isAnnotationPresent(Parameter.class)) | ||
| .filter(ConfigurationDocsGenerator::isPublicFacing) | ||
| .map(f -> toConfigurationEntry(f, configurationBean)); | ||
| } | ||
|
|
||
| private static List<Object> getDatanodeConfigurationBeans() { | ||
| return new Server().getCommandConfigurationBeans(); | ||
| } | ||
|
|
||
| /** | ||
| * There are some configuration options not intended for general usage, mainly just for system packages configuration. | ||
| * | ||
| * @see Documentation#visible() | ||
| */ | ||
| private static boolean isPublicFacing(Field f) { | ||
| return !f.isAnnotationPresent(Documentation.class) || f.getAnnotation(Documentation.class).visible(); | ||
| } | ||
|
|
||
| private static ConfigurationEntry toConfigurationEntry(Field f, Object instance) { | ||
| final String documentation = Optional.ofNullable(f.getAnnotation(Documentation.class)).map(Documentation::value).orElse(null); | ||
| final Parameter parameter = f.getAnnotation(Parameter.class); | ||
| final String propertyName = parameter.value(); | ||
| final Object defaultValue = getDefaultValue(f, instance); | ||
| final String type = getType(f); | ||
| final boolean required = parameter.required(); | ||
| return new ConfigurationEntry(instance.getClass(), f.getName(), type, propertyName, defaultValue, required, documentation); | ||
| } | ||
|
|
||
| private static Object getDefaultValue(Field f, Object instance) { | ||
| try { | ||
| return ReflectionUtils.getFieldValue(instance, f); | ||
| } catch (IllegalAccessException e) { | ||
| throw new RuntimeException(e); | ||
| } | ||
| } | ||
|
|
||
| private static String getType(Field f) { | ||
| if (f.getType().isPrimitive()) { // unify primitive types and wrappers, e.g. int -> Integer | ||
| return ClassUtils.primitiveToWrapper(f.getType()).getSimpleName(); | ||
| } else { | ||
| return f.getType().getSimpleName(); | ||
| } | ||
| } | ||
| } | ||
43 changes: 43 additions & 0 deletions
43
data-node/src/main/java/org/graylog/datanode/docs/ConfigurationEntry.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,43 @@ | ||
| /* | ||
| * Copyright (C) 2020 Graylog, Inc. | ||
| * | ||
| * This program is free software: you can redistribute it and/or modify | ||
| * it under the terms of the Server Side Public License, version 1, | ||
| * as published by MongoDB, Inc. | ||
| * | ||
| * This program is distributed in the hope that it will be useful, | ||
| * but WITHOUT ANY WARRANTY; without even the implied warranty of | ||
| * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the | ||
| * Server Side Public License for more details. | ||
| * | ||
| * You should have received a copy of the Server Side Public License | ||
| * along with this program. If not, see | ||
| * <http://www.mongodb.com/licensing/server-side-public-license>. | ||
| */ | ||
| package org.graylog.datanode.docs; | ||
|
|
||
| import jakarta.annotation.Nullable; | ||
|
|
||
| /** | ||
| * @param configurationBean Class that defines this configuration entry | ||
| * @param fieldName java field name | ||
| * @param type Java type name, e.g. String or Integer. | ||
| * @param configName configuration property name, as written in the config file | ||
| * @param defaultValue default value declared in the java field, null if not defined | ||
| * @param required if the configuration property is mandatory (needs default or entry in the config file) | ||
| * @param documentation textual documentation of this configuration propery | ||
| */ | ||
| public record ConfigurationEntry( | ||
| Class<?> configurationBean, | ||
| String fieldName, | ||
| String type, | ||
| String configName, | ||
| @Nullable Object defaultValue, | ||
| boolean required, | ||
| String documentation | ||
| ) { | ||
|
|
||
| public boolean isPriority() { | ||
| return required && defaultValue == null; | ||
| } | ||
| } |
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe we should already move that out of the generator to be able to move the generator to a server package more easily?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you have any simple idea how/where to get this?