diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md new file mode 100644 index 00000000..b3274cfc --- /dev/null +++ b/CONTRIBUTING.md @@ -0,0 +1,25 @@ +# Contributing + +Hey, thanks for your interest in contributing to this library! We welcome any +type of pull request, issues and comments! 😀 + +For pull requests, make sure you: + +* Add tests for new features and bug fixes +* Follow the existing style (4 space indent) +* Run the `clean` script before submitting a Pull Request +* Separate unrelated changes into multiple pull requests + +If you are interested in working on something, those tasks generally +fall into two categories. + +If the task you want to work on is a large task or a new task, +that is a task not recorded and prioritized in our [Issues List](/issues.md), +please [create an issue](https://github.com/jackdewinter/pymarkdown/issues), +explain the change you want to make, and include why you believe that change +is needed or is important. + +If you are not sure of what task you want to do and you just want to help, +take a look at our [Issues List](/issues.md) and see if anything grabs your +attention. If not, contact us and we can have a conversation about how you +can best contribute. diff --git a/advanced_scanning.md b/advanced_scanning.md new file mode 100644 index 00000000..47b6237d --- /dev/null +++ b/advanced_scanning.md @@ -0,0 +1,5 @@ +# Advanced Scanning + +glob support + + -l, --list-files list the markdown files found and exit \ No newline at end of file diff --git a/changelog.md b/changelog.md new file mode 100644 index 00000000..346bf66e --- /dev/null +++ b/changelog.md @@ -0,0 +1,7 @@ +# Change Log + +## Version 0.5.0 - Date: 2021-05-16 + +### Fixed and Added + +- Initial release diff --git a/docs/advanced_configuration.md b/docs/advanced_configuration.md new file mode 100644 index 00000000..cb3dab00 --- /dev/null +++ b/docs/advanced_configuration.md @@ -0,0 +1,59 @@ +# Advanced Configuration + +## Setting Configuration Values + +The configuration for this project follows a consistent theme when +deciding what configuration appplies to a given item. Sepcifically, +the order is command line setting, configuration setting, and +default setting. The special case for this ordering is the disabling +and enabling rules from the command line using the `-d` and +`---disable-rules` flags along with the `-e` and `--enable-rules` +flags. For this special case, the command line setting is +further defined as disabling a rule takes priority over enabling +a rule. While it is highly unlikely that someone will specify +both actions at the same time, we felt it was important to specify +the order to eliminate any possible confusion. + +### Command Line Setting + +### Configuration Setting + +``` + --config CONFIGURATION_FILE, -c CONFIGURATION_FILE + path to the configuration file to use +- configuration file, format, etc +``` + +## XX + +### Logging + +log.file +log.level + + --stack-trace if an error occurs, print out the stack trace for debug purposes + --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG} + minimum level required to log messages + --log-file LOG_FILE destination file for log messages + +- if debug of configuration, stack trace sets initial logging (config processing) to debug + +### Adding A Plugin + + --add-plugin ADD_PLUGIN + path to a plugin containing a new rule to apply +adding a plugin + +### Front Matter + +front matter enabled. +extensions.front-matter.enabled + +### Plugin Properties +plugins.{id}.enabled +plugins.{id}.properties + +## Configuration + +- need way of listing all plugins, info +- need way of listing all extensions diff --git a/docs/developer.md b/docs/developer.md new file mode 100644 index 00000000..0a08ae6d --- /dev/null +++ b/docs/developer.md @@ -0,0 +1,3 @@ +# Developer Documentation + +To be added. diff --git a/docs/rules.md b/docs/rules.md index c595eab3..f215873e 100644 --- a/docs/rules.md +++ b/docs/rules.md @@ -1,15 +1,15 @@ -# blah +# Implemented Rules -- [Rule Md001](xx) -- [Rule Md002](xx) -- [Rule Md003](xx) -- [Rule Md018](xx) -- [Rule Md019](xx) -- [Rule Md020](xx) -- [Rule Md021](xx) -- [Rule Md022](xx) -- [Rule Md023](xx) -- [Rule Md024](xx) -- [Rule Md026](xx) -- [Rule Md036](xx) -- [Rule Md047](xx) +- [Rule Md001 - heading-increment](rule_md001.md) +- [Rule Md002 - first-heading-h1](rule_md002.md) +- [Rule Md003 - heading-style](rule_md003.md) +- [Rule Md018 - no-missing-space-atx](rule_md018.md) +- [Rule Md019 - no-multiple-space-atx](rule_md019.md) +- [Rule Md020 - no-missing-space-closed-atx](rule_md020.md) +- [Rule Md021 - no-multiple-space-closed-atx](rule_md021.md) +- [Rule Md022 - blanks-around-headings](rule_md022.md) +- [Rule Md023 - heading-start-left](rule_md023.md) +- [Rule Md024 - no-duplicate-heading](rule_md024.md) +- [Rule Md026 - no-trailing-punctuation](rule_md026.md) +- [Rule Md036 - no-emphasis-as-heading](rule_md036.md) +- [Rule Md047 - single-trailing-newline](rule_md047.md) diff --git a/examples/example-1.md b/examples/example-1.md new file mode 100644 index 00000000..51f5d899 --- /dev/null +++ b/examples/example-1.md @@ -0,0 +1,3 @@ +## This is an example + +Just an example. \ No newline at end of file diff --git a/examples/example-2.md b/examples/example-2.md new file mode 100644 index 00000000..1a5aaed3 --- /dev/null +++ b/examples/example-2.md @@ -0,0 +1,3 @@ +# This is an example + +Just an example. diff --git a/faq.md b/faq.md new file mode 100644 index 00000000..b3d2aff6 --- /dev/null +++ b/faq.md @@ -0,0 +1,11 @@ +# Frequently Asked Questions + +## Is Markdown Front-Matter supported? + +Yes, mostly. Current support for Markdown front-matter is limited to +interpretting the front-matter as simple key-value pairs. Indenting +any following lines with 4 or more spaces causes the parser to interpret +that line as a continuation of the previous value. + +To enable this behavior in the parser, please see +[Enabling Front-Matter](efm.md). diff --git a/readme.md b/readme.md index 882c668e..b3c09b93 100644 --- a/readme.md +++ b/readme.md @@ -11,144 +11,329 @@ compliant. The rules provided in the base application can be easily extended by writing new plugins and importing them into the rules engine through simple configuration options. +The PyMarkdown project has the following advantages: + +- Consistency + - This project can example multiple files and directories with one invocation to + ensure that all detected Markdown files adhere to the same set of guidelines. +- Portable + - The linter runs on any system running Python 3.8 or later with no modifications. +- Standardized + - As the parser that powers the linter is GitHub Flavored Markdown (GFM) compliant, + it does not have to "guess" how some parsers may handle a given situation. + It follows clears rules provided by the specification on how to parse the Markdown. +- Accurate + - The parser passes all GFM conformance tests and CommonMark conformance tests. If + there was any doubt as to how a block of Markdown should be parsed, the CommonMark + 0.29.2 release was used to determine the correct parsing. +- Flexible + - Each Markdown document is parsed into an internal token format, with the preference + for writing any rules being to go against that format. Where that is not possible, + simple regular expressions and algorithms are used on a line-by-line basis. +- Thoroughly tested + - The project currently has over 2700 scenario tests and coverage percentages + all over 99%. +- Extensible + - The parser for the project adheres to the GFM specification and most of + the rules for the parser leverage the tokens produced by that parser. The + rules themselves are implemented as plugins, so they are extensible by + default. The parser itself will be extended as needed to provide for other + Markdown features as needed. + +## Note + +This project is currently in pre-release, and some of these documented things +may not work 100% as advertised until the initial release. + +## Requirements + +This project required Python 3.8 or later to function. + ## Installation ```text pip install PyMarkdown ``` -## Introduction +## How To Use -## Command Line +### If You Get Stuck -### Root Level +Full help support is available by entering -The command line help at the root level is as follows: +```shell +python main.py --help +``` -```text -usage: main.py [-h] [-e ENABLE_RULES] [-d DISABLE_RULES] [--add-plugin ADD_PLUGIN] - [--config CONFIGURATION_FILE] [--stack-trace] - [--log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG}] [--log-file LOG_FILE] - {plugins,scan,version} ... +on the command line and pressing enter. For any individual command, +help is available by following the command or commands with `--help` +as follows: -Lint any found Markdown files. +```shell +python main.py scan --help +``` -positional arguments: - {plugins,scan,version} - plugins plugin commands - scan scan the Markdown files in the specified paths - version version of the application +### Prerequisites -optional arguments: - -h, --help show this help message and exit - -e ENABLE_RULES, --enable-rules ENABLE_RULES - comma separated list of rules to enable - -d DISABLE_RULES, --disable-rules DISABLE_RULES - comma separated list of rules to disable - --add-plugin ADD_PLUGIN - path to a plugin containing a new rule to apply - --config CONFIGURATION_FILE, -c CONFIGURATION_FILE - path to the configuration file to use - --stack-trace if an error occurs, print out the stack trace for debug purposes - --log-level {CRITICAL,ERROR,WARNING,INFO,DEBUG} - minimum level required to log messages - --log-file LOG_FILE destination file for log messages +These is section requires some examples to illustrate how things work. +For the purpose of this section, this documentation will assume that +there is a file called `example-1.md` in a directory called `/examples` that +has the following content: + +```Markdown +## This is an example + +Just an example. ``` -This level of command includes three subcommands: +and a file called `example-2.md` in that same directory that has the +following content: -- [scan](dd) - - the primary workflow for this application -- [plugins](dd) - - a secondary workflow to provide information on plugins for rules -- [version](dd) - - a secondary workflow to provide information on the current version of the application +```Markdown +# This is an example -#### Notes +Just an example. -- if debug of configuration, stack trace sets initial logging (config processing) to debug +``` -### Scan +If you prefer, these files are already checked into the +[examples directory](/examples) of the GitHub project. -The command line help at the `scan` level is as follows: +### Rules -```text -usage: main.py scan [-h] [-l] path [path ...] +The PyMarkdown project includes 13 out-of-the-box [rules](/rules.md) +already implemented, with another 29 rules to be added before the +1.0.0 release. These rule are implemented using a simple plugin +system that is documented in the [developer documentation](/developer.md). +It is these rules that allow the PyMarkdown project to examine or scan +the various Markdown files, looking for particular patterns that +the authors want to consistently enforce over a set of documents. + +Note that the initial set of rules, the 42 rules provided by +David Anson's [Markdown Lint](https://github.com/markdownlint/markdownlint) +project were used as a starting point. This decision was made +to give Markdown authors that use his project in their IDEs (such as +the MarkdownLint plugin for VSCode that I use), a good grounding +in what they can consistently check for. + +### Basic Scanning + +The linter is executed by calling the project from the command line and +specifying one or more files and directories to scan for Markdown `.md` +files. The set of files and/or directories must be prefaced with the +`scan` keyword to denote that scanning is required. For the examples +directory, both this form: + +```shell +python main.py scan /examples +``` -positional arguments: - path one or more paths to scan for eligible Markdown files +and this form: -optional arguments: - -h, --help show this help message and exit - -l, --list-files list the markdown files found and exit +```shell +python main.py scan /examples/example-1.md /examples/example-2.md ``` -### Plugins +can be used to scan both file in the directory. The only difference +between the two invocations is that the first example will scan every +Markdown `.md` file in the `/examples` directory, while the second +invocation will only scan the two specified files. For clarity purposes, +if the command line specifies the same file multiple times, that file +name will only be added to the list of files to scan once. -The command line help at the `plugin` level is as follows: +### Rule Violation Format + +Executing either of the above examples will produce the following output: ```text -usage: main.py plugins [-h] {list,info} ... +/examples/example-1.md:3:16: MD047: Each file should end with a single +newline character. (single-trailing-newline) +``` + +The format of the output for any rules that are triggered is as follows: + +`file-name:line:column: rule-id: description (aliases)` + +- `file-name` - Path to the file that triggered the rule. +- `line`/`column` - Position in the file where the rule was triggered +- `rule-id` - Unique identifier assigned to the rule. +- `description` - Human readable description of the rule. +- `aliases` - One or more aliases used to reference the rule. + +For this example and the rule violation that was reported, the first +step is to look at the file `/examples/example-1.md` at the end of +line 3, which is column 16. Rule [md047](\docs\rule_md047.md) specifies +that every file should end with a single newline character, which is +what is reported in the description. Additionally, it reports that this +rule can also be identified by the more human readable alias of +`single-trailing-newline`. + +### Advanced Scanning -positional arguments: - {list,info} - list list the available plugins - info information of specific plugins +For more advanced scanning options, please consult the document +on [Advanced Scanning](advanced_scanning.md). -optional arguments: - -h, --help show this help message and exit +### Plugin Rule Information + +For information on what rules are currently present, the following +command may be used: + +```shell +python main.py plugins list ``` -This level of command includes three subcommands: +This command will list all of the rules in a table in the following format: -- [list](dd) - - a workflow to list every rule plugin that is registered with the application -- [info](dd) - - a workflow to provide information on a single rule plugin +`rule-id aliases enabled-default enabled-current version` -### List +- `rule-id` - Unique identifier assigned to the rule. +- `aliases` - One or more aliases used to reference the rule. +- `enabled-default` - Whether the rule is enabled by default. +- `enabled-current` - Whether the rule is currently enabled. +- `version` - Version associated with the rule. If the rule is a project + rule, this version will always be the version of the project. -The command line help at the `list` level is as follows: +In addition, the `list` command may be followed by text that +specifies a Glob pattern used to match against the rules. For example, +using the command `plugins list md00?` produced this output: ```text -usage: main.py plugins list [-h] [list_filter] +ID NAMES ENABLED (DEFAULT) ENABLED (CURRENT) VERSION + +md047 first-heading-h1, first False False 0.5.0 + -header-h1 +``` -positional arguments: - list_filter filter +If more verbose information is needed on a given rule, the +`plugins info` command may be used with a specific `rule-id` +or `alias` used to refer to the plugin. If provided with a `rule-id` +of `md047` or `single-trailing-newline`, this command will produce +the following output: -optional arguments: - -h, --help show this help message and exit +```text +Id:md047 +Name(s):single-trailing-newline +Description:Each file should end with a single newline character. ``` -### Info +- Note that better support for this command is priortized as + required for the general release and should happen fairly quickly -The command line help at the `info` level is as follows: +### Basic Configuration -```text -usage: main.py plugins info [-h] info_filter +The most frequently used part of the configuration system is the +part that enables and disables specific rules while scanning the +Markdown files. For example, if you do not like rule md047 which +states that each file must end with a single newline, you can +disable that rule by specifying: -positional arguments: - info_filter an id +```shell +python main.py -d md047 scan /examples +``` + +or: + +```shell +python main.py --disable-rules md047 scan /examples +``` -optional arguments: - -h, --help show this help message and exit +The effect of disabling the rule should be evidenced by +the scan no longer reporting any violations of rule md047 +against the Markdown file `example-1.md`. + +Alternatively, rules can also be enabled. As the modelled +base rules for this project are based off of those for David +Anson's project, rule md002 is disable by default in both +projects. Specifically, rule md002 is disabled by default +as rule md041 provides a better implementation of that rule +that takes front-matter into account. Until that rule is +implemented, you can enable rule md002 by specifying either: + +```shell +python main.py -e md002 scan /examples +``` + +or + +```shell +python main.py --enable-rules md002 scan /examples +``` + +The effect of enabling the rule should be evidenced by +the scan reporting a violation of rule md002 against +Markdown file `example-1.md`: + +```text +examples/example-1.md:1:1: MD002: First heading of the document should +be a top level heading. [Expected: h1; Actual: h2] (first-heading-h1, +first-header-h1) +examples/example-1.md:3:16: MD047: Each file should end with a single +newline character. (single-trailing-newline) ``` -### Version +### Advanced Configuration + +For more advanced configuration options, please consult the document +on [Advanced Configuration](advanced_configuration.md). This +document includes information on: + +- specifying configuration files + +## Open Issues and Future Plans + +During the development phase of this project, it was more useful to have +an actual list of issues to track and prioritize, rather than relying on +GitHub to do all of the work. Here is the [Issues List](/issues.md). + +If you find any issues, please report them using the standard GitHub +issues process. When our team takes a look at your issue and triages +it, it will be added to our Issues List with the specified priority. +For us, this provides transparency as to what we are currently working +on, what is up next, and what our future plans are. + +## Still Have Questions? + +If you still have questions, please consult our +[Frequently Asked Questions](faq.md) document. + +## Version Information + +The changelog for this project is maintained [at this location](/changelog.md). + +## Contact Information + +If you would like to report an issue with the linter, a rule, or +the documentation, please file an issue [using GitHub](issues.md). + +If you would like to help fix a specific issue or do some work to +implement a feature that you believe is important, please file +an issue that includes what you want to add, why you want to add +it, and why it is important. + +If you would like to contribute to the project in a more +substantial manner, please contact me at `jack.de.winter@outlook.com`. -This command emits a single line that show the version of the application. +## Instructions For Contributing -## Configuration +See [CONTRIBUTING.md](/CONTRIBUTING.md) file. -log.file -log.level -plugins.{id}.enabled -plugins.{id}.properties -extensions.front-matter.enabled +## Acknowledgements -plugin ordering: command line (disabled, enabled), config, default -others ordering: command line (if exposed), config, default +Currently, as a team of one, there are only two big groups of people to +acknowledge. -- need way of listing all plugins, info -- need way of listing all extensions +The first, and foremost group, is my immediate family. They +have endured me coming out of my office with my head still in the +clouds, explaining things to them so that I can think more clearly. +While they still do not understand what I am talking about with respect +to this project, I am so grateful to them for allowing me to work "my +process" to figure things out. +The second group is the contributors to the +[CommonMark discussion forum](https://talk.commonmark.org/). +While I have raised some issues that were cut and dry, a lot of them +involved significant amount of discussion to figure out what the +right approach is. Through all those discussions, I rarely, if ever, +felt like they treated me as less than equal, no matter how stupid +my questions was. For their patience and their professionalism, +thank you.