From 56d5ded7139fa010db6cf93eb966bc3962c4c96d Mon Sep 17 00:00:00 2001 From: Laurence Date: Tue, 30 Sep 2025 11:24:06 +0100 Subject: [PATCH 1/4] docs: enhance hub contributing documentation with comprehensive guidelines - Add communication channels section (Discord, Discourse, GitHub) - Expand testing section with accurate cscli hubtest workflow - Add YAML best practices with real-world anchor examples - Include AI-assisted generation guidelines with disclosure requirements - Align PR checklist with enforced GitHub PR template - Add troubleshooting section with common issues and solutions - Consolidate redundant "Getting Started" and "Getting Involved" sections - Fix MDX parsing issues and improve markdown formatting - Add comprehensive git workflow and branch naming conventions - Include detailed documentation requirements and examples --- crowdsec-docs/docs/contributing/hub.md | 440 ++++++++++++++++++++++--- 1 file changed, 402 insertions(+), 38 deletions(-) diff --git a/crowdsec-docs/docs/contributing/hub.md b/crowdsec-docs/docs/contributing/hub.md index d54eafea0..c1d06d803 100644 --- a/crowdsec-docs/docs/contributing/hub.md +++ b/crowdsec-docs/docs/contributing/hub.md @@ -4,26 +4,275 @@ title: Hub sidebar_position: 3 --- -Parsers, Scenarios, Collections allow the `Security Engine` to detect and block malevolent behavior. +# Contributing to the Hub -Supporting new services or improving the detection capabilities on existing software is a great way to contribute. +Parsers, Scenarios, and Collections allow the CrowdSec `Security Engine` to detect and block malevolent behavior. Supporting new services or improving the detection capabilities on existing software is a great way to contribute to the CrowdSec ecosystem. Sharing your parsers, scenarios and collections on the hub allows other users to use them to protect themselves. -# Contributing +## Communication -Anyone can open an issue about parsers/scenarios, or contribute a change with a pull request (PR) to the crowdsecuity/hub GitHub repository. You need to be comfortable with git and GitHub to work effectively. +The main communication channels for hub contributions are: -To get involved : +- [Discord](https://discord.gg/crowdsec): Best for live interactions and quick questions about hub development +- [Discourse](https://discourse.crowdsec.net/): Great for discussing ideas, suggesting improvements, or asking detailed questions +- [GitHub Issues](https://github.com/crowdsecurity/hub/issues): Use for bug reports and feature requests +- [GitHub Discussions](https://github.com/crowdsecurity/hub/discussions): For general questions and community discussions -- Have a look at open [issues](https://github.com/crowdsecurity/hub/issues) and [pull requests](https://github.com/crowdsecurity/hub/pulls) -- Clone the hub repository -- Create/Modify parsers/scenarios/collections -- Create/Modify tests to ensure proper coverage -- Open a pull request +## Getting Started + +Anyone can open an issue about parsers/scenarios, or contribute a change with a pull request (PR) to the [crowdsecurity/hub](https://github.com/crowdsecurity/hub) GitHub repository. You need to be comfortable with git and GitHub to work effectively. + +### Find something to work on + +Here are some things you can do today to start contributing: + +- Help improve existing parsers and scenarios +- Add support for new services and applications +- Create comprehensive test coverage +- Help triage issues and review pull requests +- Improve documentation and examples + +### Find a good first topic + +The hub repository has beginner-friendly issues that are a great place to get started: + +- Look for issues labeled \`good first issue\` - these don't require high-level CrowdSec knowledge +- Issues labeled \`help wanted\` indicate that community help is particularly welcome + +### Prerequisites + +Before contributing, make sure you have: + +- Basic understanding of YAML syntax +- Familiarity with grok patterns (for parsers) +- Git and GitHub knowledge +- A local CrowdSec installation for testing (optional but recommended) + +### Contribution Workflow + +The basic workflow for contributing: + +1. Have a look at open [issues](https://github.com/crowdsecurity/hub/issues) and [pull requests](https://github.com/crowdsecurity/hub/pulls) +2. Fork and clone the hub repository +3. Create/Modify parsers/scenarios/collections +4. Create/Modify tests to ensure proper coverage +5. Open a pull request + +## Testing + +Before submitting your contribution, ensure proper testing using `cscli hubtest`: + +### Local Testing with cscli hubtest + +`cscli hubtest` is the primary tool for testing hub components. It creates tests for parsers, scenarios, and collections. + +#### 1. Create a test + +```bash +# Create a test for a parser +cscli hubtest create my-parser-test --type syslog + +# Create a test for a scenario (skip parser testing) +cscli hubtest create my-scenario-test --type syslog --ignore-parsers + +# Create a test for specific components +cscli hubtest create my-test --parsers crowdsecurity/nginx --scenarios crowdsecurity/http-probing +``` + +#### 2. Configure your test + +Edit the generated configuration file (\`.tests/<test-name>/config.yaml\`): + +```yaml +parsers: + - crowdsecurity/syslog-logs + - ./parsers/s01-parse/crowdsecurity/my-parser.yaml +scenarios: + - ./scenarios/crowdsecurity/my-scenario.yaml +postoverflows: +log_file: my-test.log +log_type: syslog +ignore_parsers: false +``` + +#### 3. Add test data and assertions + +- **Log file**: Add sample logs to \`.tests/<test-name>/<test-name>.log\` +- **Parser assertions**: Define expected parsed fields in \`parser.assert\` +- **Scenario assertions**: Define expected alerts in \`scenario.assert\` + +**Note**: When you first run \`cscli hubtest run\`, it will output the generated assertions that you need to fill out in \`parser.assert\` or \`scenario.assert\` files. You can find examples of assertion files in the hub repository at \`.tests/<existing-test>/parser.assert\` and \`.tests/<existing-test>/scenario.assert\`. + +#### 4. Run your test + +```bash +# Run a specific test +cscli hubtest run my-parser-test + +# Run all tests +cscli hubtest run --all + +# Run with verbose output +cscli hubtest run my-parser-test -v +``` + +#### 5. Debug your test + +```bash +# Explain test results +cscli hubtest explain my-parser-test + +# Show only failures +cscli hubtest explain my-parser-test --failures + +# Verbose explanation +cscli hubtest explain my-parser-test -v +``` + +### CI/CD Testing + +The hub repository uses GitHub Actions for automated testing: + +- All parsers and scenarios are automatically tested when you open a PR +- Tests include syntax validation, pattern matching, and integration tests +- Make sure all tests pass before requesting review + +### Test Coverage + +Check test coverage for your contributions: + +```bash +cscli hubtest coverage +``` + +## Git Workflow / Branch Management + +We receive contributions on the \`master\` branch. To contribute: + +1. **Fork the repository** on GitHub +2. **Clone your fork** locally: + ```bash + git clone https://github.com/yourusername/hub.git + cd hub + ``` +3. **Create a dedicated branch** for your contribution: + ```bash + git checkout -b feature/your-feature-name + ``` +4. **Make your changes** and commit them with descriptive messages +5. **Push to your fork**: + ```bash + git push origin feature/your-feature-name + ``` +6. **Open a Pull Request** targeting the \`master\` branch + +### Branch Naming Convention + +Use descriptive branch names that indicate the type of contribution: +- \`feature/parser-nginx-access-logs\` +- \`fix/scenario-ssh-bruteforce-labels\` +- \`docs/collection-apache-examples\` + +### Commit Messages + +Write clear, descriptive commit messages: +- Use imperative mood: "Add parser for Apache access logs" +- Reference issues when applicable: "Fix scenario labels (#123)" +- Keep the first line under 50 characters ## Guidelines +### YAML Best Practices + +When creating parsers, scenarios, and collections, follow these YAML guidelines: + +#### Avoid YAML Anchors + +**❌ Don't use YAML anchors** - they make YAML files harder to maintain and understand: + +```yaml +# BAD - Using anchors in nodes +nodes: +- grok: + pattern: 'pattern1' + apply_on: message + nodes: &message_parsers + - grok: + pattern: "user pattern" + apply_on: parsedmessage + statics: + - meta: sub_type + value: user_enumeration +- grok: + pattern: 'pattern2' + apply_on: message + nodes: *message_parsers +``` + +**✅ Duplicate the node structure instead of using anchors:** + +```yaml +# GOOD - No anchors, explicit node definitions +nodes: +- grok: + pattern: 'pattern1' + apply_on: message + nodes: + - grok: + pattern: "user pattern" + apply_on: parsedmessage + statics: + - meta: sub_type + value: user_enumeration +- grok: + pattern: 'pattern2' + apply_on: message + nodes: + - grok: + pattern: "user pattern" + apply_on: parsedmessage + statics: + - meta: sub_type + value: user_enumeration +``` + +#### Benefits of Using Statics + +Yes, we understand that using YAML anchors makes the YAML DRY but it can introduce complexity. + +- **Better maintainability**: Static values are clearly defined at the top level +- **Automatic key handling**: If a static returns empty data, the key will not be set (as you mentioned) +- **Cleaner structure**: Avoids YAML complexity and improves readability +- **Consistent patterns**: Follows established CrowdSec conventions +- **Proper targeting**: Statics can target \`meta\`, \`parsed\`, or \`enriched\` fields directly +- **Expression support**: Can use expressions to derive values dynamically + +### AI-Assisted Generation + +We do allow AI-assisted generation of parsers, scenarios, and collections, but with important requirements: + +#### Requirements for AI-Generated Contributions + +- **Follow all guidelines**: AI-generated code must still follow all the guidelines in this document +- **Include comprehensive tests**: A pull request with AI-generated code but no tests will be immediately closed +- **Proper documentation**: Include complete documentation and examples as required +- **Code quality**: Ensure the generated code follows CrowdSec conventions and best practices + +#### Disclosure Requirement + +**Important**: When submitting AI-assisted contributions, you must check the "AI was used to generate any/all content of this PR" box in the PR template. You must understand and be able to explain yourself what was generated. AI-generated code will receive additional scrutiny to ensure quality and correctness. + +#### What We Expect + +- Test coverage using \`cscli hubtest\` +- Proper error handling and edge cases +- Clear documentation and examples +- Adherence to CrowdSec patterns and conventions +- Human review and validation of the AI output + +AI is a powerful tool, but it should augment human expertise, not replace proper testing and review processes. + ### Technical Documentation The following explains how to create and test: @@ -33,28 +282,49 @@ The following explains how to create and test: ### Collections -It often makes sense for a new parser or scenario to be added to an existing [collection](/log_processor/collections/format.md), or create a new one. +Collections group related parsers, scenarios, and postoverflows together. It often makes sense for a new parser or scenario to be added to an existing [collection](/log_processor/collections/format.md), or create a new one. + +#### When to create a new collection: + +- Your parsers and/or scenarios cover a new or specific service +- You're adding multiple related components that work together +- The existing collections don't fit your use case + +#### When to add to existing collections: + +- Adding a parser for \`SpecificWebServer\` access logs that would benefit from [all the default http related scenarios](https://hub.crowdsec.net/author/crowdsecurity/collections/base-http-scenarios) +- Your contribution enhances an existing service's detection capabilities +- Your scenario complements existing parsers in a collection -If your parsers and/or scenarios cover a new or specific service, having a dedicated collection for this service makes sense. -In other cases, having a parser for `SpecificWebServer` access logs would justify a collection as it might also include [all the default http related scenarios](https://hub.crowdsec.net/author/crowdsecurity/collections/base-http-scenarios). +#### Collection structure: + +Each collection should include: +- Descriptive name and description +- Proper labels and tags +- Documentation with setup instructions +- Example acquisition configuration +- Related parsers, scenarios, and postoverflows ### Scenarios -When you create a scenario, you must fill some fields in the [`labels`](/log_processor/scenarios/format.md#labels), else the CI won't accept the contribution. -Those `labels` are: - - `classification`: this array contains the CVE ID and the [Mitre Techniques](https://attack.mitre.org/techniques/enterprise/) related to the scenario (when applicable) - - `spoofable`: between 0 and 3, is the chance that the attacker behind the attack can spoof its origin - - `confidence`: between 0 and 3, is the confidence that the scenario will not trigger false positive - - `behaviors`: an existing behavior in [this file](https://github.com/crowdsecurity/hub/blob/scenario_taxonomy/taxonomy/behaviors.json) - - `label` : a human readable name for the scenario - - `cti` : (optional) true or false, used to specify that a scenario is mainly used for audit rather than detecting a threat +Scenarios define the logic for detecting attacks and suspicious behavior. When you create a scenario, you must fill some fields in the [\`labels\`](/log_processor/scenarios/format.md#labels), else the CI won't accept the contribution. -[Here](/log_processor/scenarios/format.md#labels) is the `labels` documentation for more information. +#### Required Labels -Here is an example: +Those \`labels\` are: + - \`classification\`: this array contains the CVE ID and the [Mitre Techniques](https://attack.mitre.org/techniques/enterprise/) related to the scenario (when applicable) + - \`spoofable\`: between 0 and 3, is the chance that the attacker behind the attack can spoof its origin + - \`confidence\`: between 0 and 3, is the confidence that the scenario will not trigger false positive + - \`behaviors\`: an existing behavior in [this file](https://github.com/crowdsecurity/hub/blob/scenario_taxonomy/taxonomy/behaviors.json) + - \`label\` : a human readable name for the scenario + - \`cti\` : (optional) true or false, used to specify that a scenario is mainly used for audit rather than detecting a threat -``` +[Here](/log_processor/scenarios/format.md#labels) is the \`labels\` documentation for more information. + +#### Example Scenario Labels + +```yaml labels: service: ssh confidence: 3 @@ -66,30 +336,124 @@ labels: remediation: true ``` +#### Label Guidelines + +- **Confidence**: Start with 3 (high confidence) and reduce if you're unsure +- **Spoofable**: Consider if the attack source can be easily faked (0 = cannot be spoofed, 3 = easily spoofed) +- **Classification**: Use MITRE ATT&CK techniques when applicable +- **Behavior**: Choose from the predefined taxonomy to ensure consistency + ## Preparing your contribution Before asking for a review of your PR, please ensure you have the following: -- tests: Test creation is covered in [parsers creation](/log_processor/parsers/create.md) and [scenarios creation](/log_processor/scenarios/create.md). Ensure that each of your parser or scenario is properly tested. -- documentation: Please provide a `.md` file with the same name as each of your parser, scenario or collection. The markdown is rendered in the [hub](https://hub.crowdsec.net). -- documentation: If you're creating a collection targeting a specific log file, be sure to provide an acquis example as : +### Tests +Test creation is covered in [parsers creation](/log_processor/parsers/create.md) and [scenarios creation](/log_processor/scenarios/create.md). Ensure that each of your parser or scenario is properly tested: -```yaml +- **Parser tests**: Include sample log files that cover various scenarios (success, failure, edge cases) +- **Scenario tests**: Test with different input data to ensure proper triggering and no false positives +- **Integration tests**: Verify that your components work well with existing parsers/scenarios + +### Documentation + +Please provide a `.md` file with the same name as each of your parser, scenario or collection. The markdown is rendered in the [hub](https://hub.crowdsec.net). + +#### Documentation Requirements - ## Acquisition template +- Clear description of what the component does +- Setup instructions and prerequisites +- Configuration examples +- Troubleshooting tips +- Links to relevant documentation - Example acquisition for this collection : +#### Collection Documentation - ```yaml - filenames: - - /var/log/xxx/*.log - labels: - type: something - ``` +If you're creating a collection targeting a specific log file, be sure to provide an acquisition example: + +```yaml +filenames: +- /var/log/xxx/*.log +labels: + type: something ``` -## Open your PR +### Code Quality + +- Follow existing naming conventions +- Use consistent indentation and formatting +- Add comments for complex logic +- Ensure all required fields are properly filled + +## Opening your PR + +Everything is all set, you can now open a PR that will be reviewed and merged! + +### PR Checklist + +Before opening your PR, ensure you can check all items in the [PR template](https://github.com/crowdsecurity/hub/blob/master/.github/pull_request_template.md): + +- [ ] I have read the [contributing guide](https://docs.crowdsec.net/docs/next/contributing/contributing_hub) +- [ ] I have tested my changes locally +- [ ] For new parsers or scenarios, tests have been added +- [ ] I have run the hub linter and no issues were reported (see contributing guide) +- [ ] Automated tests are passing +- [ ] AI was used to generate any/all content of this PR (if applicable) + +#### Additional Requirements + +- [ ] Documentation is complete and accurate +- [ ] Code follows the project's style guidelines +- [ ] Commit messages are clear and descriptive +- [ ] PR description explains the changes and motivation +- [ ] Related issues are referenced (if applicable) + +### Review Process + +- PRs are reviewed by maintainers and community members +- Feedback may be requested for improvements +- All CI checks must pass before merging +- Maintainers will merge approved PRs + +## Troubleshooting + +### Common Issues + +#### Parser Issues + +- **Grok patterns not matching**: Use online grok testers to validate patterns +- **Missing fields**: Ensure all required fields are extracted +- **Performance**: Optimize grok patterns for better performance + +#### Scenario Issues + +- **False positives**: Adjust thresholds and conditions +- **Not triggering**: Check that events are properly parsed and available +- **Label validation**: Ensure all required labels are present and valid + +#### Testing Issues + +- **Hubtest creation fails**: Ensure you're in the hub repository root directory +- **Test configuration errors**: Check YAML syntax in \`.tests/<test-name>/config.yaml\` +- **Parser assertions failing**: Use \`cscli hubtest explain <test-name>\` to debug parser output +- **Scenario assertions failing**: Verify scenario logic and thresholds with \`cscli hubtest explain <test-name>\` +- **Missing test data**: Provide comprehensive log samples and assertion files +- **CI tests failing**: Review the GitHub Actions logs for specific errors + +### Getting Help + +If you encounter issues: + +1. Check existing [GitHub Issues](https://github.com/crowdsecurity/hub/issues) for similar problems +2. Ask for help on [Discord](https://discord.gg/crowdsec) or [Discourse](https://discourse.crowdsec.net/) +3. Open a new issue with detailed information about your problem + +### Useful Resources -Everything is all set, you can now open a PR, that will be reviewed and merged! +- [Parser Creation Guide](/log_processor/parsers/create.md) +- [Scenario Creation Guide](/log_processor/scenarios/create.md) +- [Collection Format](/log_processor/collections/format.md) +- [Expression Language Documentation](/expr/intro.md) +- [cscli hubtest Documentation](/cscli/cscli_hubtest.md) +- [Hub Website](https://hub.crowdsec.net) From a04917c2bc4b960f85aae48ef70617f1e2e0b6f2 Mon Sep 17 00:00:00 2001 From: Laurence Date: Tue, 30 Sep 2025 11:27:28 +0100 Subject: [PATCH 2/4] fix: remove unnecessary backtick escaping in markdown - Fix inline code backticks that were over-escaped - Keep proper escaping only for backticks inside code blocks - Improve markdown readability and rendering --- crowdsec-docs/docs/contributing/hub.md | 54 +++++++++++++------------- 1 file changed, 27 insertions(+), 27 deletions(-) diff --git a/crowdsec-docs/docs/contributing/hub.md b/crowdsec-docs/docs/contributing/hub.md index c1d06d803..6db062f7f 100644 --- a/crowdsec-docs/docs/contributing/hub.md +++ b/crowdsec-docs/docs/contributing/hub.md @@ -37,8 +37,8 @@ Here are some things you can do today to start contributing: The hub repository has beginner-friendly issues that are a great place to get started: -- Look for issues labeled \`good first issue\` - these don't require high-level CrowdSec knowledge -- Issues labeled \`help wanted\` indicate that community help is particularly welcome +- Look for issues labeled `good first issue` - these don't require high-level CrowdSec knowledge +- Issues labeled `help wanted` indicate that community help is particularly welcome ### Prerequisites @@ -82,7 +82,7 @@ cscli hubtest create my-test --parsers crowdsecurity/nginx --scenarios crowdsecu #### 2. Configure your test -Edit the generated configuration file (\`.tests/<test-name>/config.yaml\`): +Edit the generated configuration file (`.tests//config.yaml`): ```yaml parsers: @@ -98,11 +98,11 @@ ignore_parsers: false #### 3. Add test data and assertions -- **Log file**: Add sample logs to \`.tests/<test-name>/<test-name>.log\` -- **Parser assertions**: Define expected parsed fields in \`parser.assert\` -- **Scenario assertions**: Define expected alerts in \`scenario.assert\` +- **Log file**: Add sample logs to `.tests//.log` +- **Parser assertions**: Define expected parsed fields in `parser.assert` +- **Scenario assertions**: Define expected alerts in `scenario.assert` -**Note**: When you first run \`cscli hubtest run\`, it will output the generated assertions that you need to fill out in \`parser.assert\` or \`scenario.assert\` files. You can find examples of assertion files in the hub repository at \`.tests/<existing-test>/parser.assert\` and \`.tests/<existing-test>/scenario.assert\`. +**Note**: When you first run `cscli hubtest run`, it will output the generated assertions that you need to fill out in `parser.assert` or `scenario.assert` files. You can find examples of assertion files in the hub repository at `.tests//parser.assert` and `.tests//scenario.assert`. #### 4. Run your test @@ -148,7 +148,7 @@ cscli hubtest coverage ## Git Workflow / Branch Management -We receive contributions on the \`master\` branch. To contribute: +We receive contributions on the `master` branch. To contribute: 1. **Fork the repository** on GitHub 2. **Clone your fork** locally: @@ -165,14 +165,14 @@ We receive contributions on the \`master\` branch. To contribute: ```bash git push origin feature/your-feature-name ``` -6. **Open a Pull Request** targeting the \`master\` branch +6. **Open a Pull Request** targeting the `master` branch ### Branch Naming Convention Use descriptive branch names that indicate the type of contribution: -- \`feature/parser-nginx-access-logs\` -- \`fix/scenario-ssh-bruteforce-labels\` -- \`docs/collection-apache-examples\` +- `feature/parser-nginx-access-logs` +- `fix/scenario-ssh-bruteforce-labels` +- `docs/collection-apache-examples` ### Commit Messages @@ -245,7 +245,7 @@ Yes, we understand that using YAML anchors makes the YAML DRY but it can introdu - **Automatic key handling**: If a static returns empty data, the key will not be set (as you mentioned) - **Cleaner structure**: Avoids YAML complexity and improves readability - **Consistent patterns**: Follows established CrowdSec conventions -- **Proper targeting**: Statics can target \`meta\`, \`parsed\`, or \`enriched\` fields directly +- **Proper targeting**: Statics can target `meta`, `parsed`, or `enriched` fields directly - **Expression support**: Can use expressions to derive values dynamically ### AI-Assisted Generation @@ -265,7 +265,7 @@ We do allow AI-assisted generation of parsers, scenarios, and collections, but w #### What We Expect -- Test coverage using \`cscli hubtest\` +- Test coverage using `cscli hubtest` - Proper error handling and edge cases - Clear documentation and examples - Adherence to CrowdSec patterns and conventions @@ -292,7 +292,7 @@ Collections group related parsers, scenarios, and postoverflows together. It oft #### When to add to existing collections: -- Adding a parser for \`SpecificWebServer\` access logs that would benefit from [all the default http related scenarios](https://hub.crowdsec.net/author/crowdsecurity/collections/base-http-scenarios) +- Adding a parser for `SpecificWebServer` access logs that would benefit from [all the default http related scenarios](https://hub.crowdsec.net/author/crowdsecurity/collections/base-http-scenarios) - Your contribution enhances an existing service's detection capabilities - Your scenario complements existing parsers in a collection @@ -308,19 +308,19 @@ Each collection should include: ### Scenarios -Scenarios define the logic for detecting attacks and suspicious behavior. When you create a scenario, you must fill some fields in the [\`labels\`](/log_processor/scenarios/format.md#labels), else the CI won't accept the contribution. +Scenarios define the logic for detecting attacks and suspicious behavior. When you create a scenario, you must fill some fields in the [`labels`](/log_processor/scenarios/format.md#labels), else the CI won't accept the contribution. #### Required Labels -Those \`labels\` are: - - \`classification\`: this array contains the CVE ID and the [Mitre Techniques](https://attack.mitre.org/techniques/enterprise/) related to the scenario (when applicable) - - \`spoofable\`: between 0 and 3, is the chance that the attacker behind the attack can spoof its origin - - \`confidence\`: between 0 and 3, is the confidence that the scenario will not trigger false positive - - \`behaviors\`: an existing behavior in [this file](https://github.com/crowdsecurity/hub/blob/scenario_taxonomy/taxonomy/behaviors.json) - - \`label\` : a human readable name for the scenario - - \`cti\` : (optional) true or false, used to specify that a scenario is mainly used for audit rather than detecting a threat +Those `labels` are: + - `classification`: this array contains the CVE ID and the [Mitre Techniques](https://attack.mitre.org/techniques/enterprise/) related to the scenario (when applicable) + - `spoofable`: between 0 and 3, is the chance that the attacker behind the attack can spoof its origin + - `confidence`: between 0 and 3, is the confidence that the scenario will not trigger false positive + - `behaviors`: an existing behavior in [this file](https://github.com/crowdsecurity/hub/blob/scenario_taxonomy/taxonomy/behaviors.json) + - `label` : a human readable name for the scenario + - `cti` : (optional) true or false, used to specify that a scenario is mainly used for audit rather than detecting a threat -[Here](/log_processor/scenarios/format.md#labels) is the \`labels\` documentation for more information. +[Here](/log_processor/scenarios/format.md#labels) is the `labels` documentation for more information. #### Example Scenario Labels @@ -435,9 +435,9 @@ Before opening your PR, ensure you can check all items in the [PR template](http #### Testing Issues - **Hubtest creation fails**: Ensure you're in the hub repository root directory -- **Test configuration errors**: Check YAML syntax in \`.tests/<test-name>/config.yaml\` -- **Parser assertions failing**: Use \`cscli hubtest explain <test-name>\` to debug parser output -- **Scenario assertions failing**: Verify scenario logic and thresholds with \`cscli hubtest explain <test-name>\` +- **Test configuration errors**: Check YAML syntax in `.tests//config.yaml` +- **Parser assertions failing**: Use `cscli hubtest explain ` to debug parser output +- **Scenario assertions failing**: Verify scenario logic and thresholds with `cscli hubtest explain ` - **Missing test data**: Provide comprehensive log samples and assertion files - **CI tests failing**: Review the GitHub Actions logs for specific errors From 49d905fbb4ed6c892bf47a0b41d5c7d5bdfa322b Mon Sep 17 00:00:00 2001 From: Laurence Date: Tue, 30 Sep 2025 11:28:25 +0100 Subject: [PATCH 3/4] enhance: remove discussions --- crowdsec-docs/docs/contributing/hub.md | 1 - 1 file changed, 1 deletion(-) diff --git a/crowdsec-docs/docs/contributing/hub.md b/crowdsec-docs/docs/contributing/hub.md index 6db062f7f..1f1211ac9 100644 --- a/crowdsec-docs/docs/contributing/hub.md +++ b/crowdsec-docs/docs/contributing/hub.md @@ -17,7 +17,6 @@ The main communication channels for hub contributions are: - [Discord](https://discord.gg/crowdsec): Best for live interactions and quick questions about hub development - [Discourse](https://discourse.crowdsec.net/): Great for discussing ideas, suggesting improvements, or asking detailed questions - [GitHub Issues](https://github.com/crowdsecurity/hub/issues): Use for bug reports and feature requests -- [GitHub Discussions](https://github.com/crowdsecurity/hub/discussions): For general questions and community discussions ## Getting Started From 23937cd263863b09a24875a22d0315d8634c3136 Mon Sep 17 00:00:00 2001 From: Laurence Date: Tue, 30 Sep 2025 13:54:56 +0100 Subject: [PATCH 4/4] fix: address colleague feedback on hub contributing docs - Remove cscli hubtest coverage command that shows 0% and lacks filtering - Remove ChatGPT remnants and unclear language - Simplify PR checklist to avoid template duplication - Remove vague performance troubleshooting item - Remove redundant benefits section for cleaner flow --- crowdsec-docs/docs/contributing/hub.md | 32 ++------------------------ 1 file changed, 2 insertions(+), 30 deletions(-) diff --git a/crowdsec-docs/docs/contributing/hub.md b/crowdsec-docs/docs/contributing/hub.md index 1f1211ac9..9c026731c 100644 --- a/crowdsec-docs/docs/contributing/hub.md +++ b/crowdsec-docs/docs/contributing/hub.md @@ -137,13 +137,6 @@ The hub repository uses GitHub Actions for automated testing: - Tests include syntax validation, pattern matching, and integration tests - Make sure all tests pass before requesting review -### Test Coverage - -Check test coverage for your contributions: - -```bash -cscli hubtest coverage -``` ## Git Workflow / Branch Management @@ -236,17 +229,6 @@ nodes: value: user_enumeration ``` -#### Benefits of Using Statics - -Yes, we understand that using YAML anchors makes the YAML DRY but it can introduce complexity. - -- **Better maintainability**: Static values are clearly defined at the top level -- **Automatic key handling**: If a static returns empty data, the key will not be set (as you mentioned) -- **Cleaner structure**: Avoids YAML complexity and improves readability -- **Consistent patterns**: Follows established CrowdSec conventions -- **Proper targeting**: Statics can target `meta`, `parsed`, or `enriched` fields directly -- **Expression support**: Can use expressions to derive values dynamically - ### AI-Assisted Generation We do allow AI-assisted generation of parsers, scenarios, and collections, but with important requirements: @@ -291,7 +273,7 @@ Collections group related parsers, scenarios, and postoverflows together. It oft #### When to add to existing collections: -- Adding a parser for `SpecificWebServer` access logs that would benefit from [all the default http related scenarios](https://hub.crowdsec.net/author/crowdsecurity/collections/base-http-scenarios) +- Adding a parser for a specific web server's access logs that would benefit from existing HTTP-related scenarios - Your contribution enhances an existing service's detection capabilities - Your scenario complements existing parsers in a collection @@ -391,16 +373,7 @@ Everything is all set, you can now open a PR that will be reviewed and merged! ### PR Checklist -Before opening your PR, ensure you can check all items in the [PR template](https://github.com/crowdsecurity/hub/blob/master/.github/pull_request_template.md): - -- [ ] I have read the [contributing guide](https://docs.crowdsec.net/docs/next/contributing/contributing_hub) -- [ ] I have tested my changes locally -- [ ] For new parsers or scenarios, tests have been added -- [ ] I have run the hub linter and no issues were reported (see contributing guide) -- [ ] Automated tests are passing -- [ ] AI was used to generate any/all content of this PR (if applicable) - -#### Additional Requirements +Before opening your PR, ensure you can check all items in the [PR template](https://github.com/crowdsecurity/hub/blob/master/.github/pull_request_template.md). Additional requirements: - [ ] Documentation is complete and accurate - [ ] Code follows the project's style guidelines @@ -423,7 +396,6 @@ Before opening your PR, ensure you can check all items in the [PR template](http - **Grok patterns not matching**: Use online grok testers to validate patterns - **Missing fields**: Ensure all required fields are extracted -- **Performance**: Optimize grok patterns for better performance #### Scenario Issues