Skip to content

Commit

Permalink
docs: introduce "curldown" for libcurl man page format
Browse files Browse the repository at this point in the history
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
  their man page section is specified)

tools:

- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown

This setup generates .3 versions of all the curldown versions at build time.

CI:

Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...

Closes curl#12730
  • Loading branch information
bagder committed Jan 22, 2024
1 parent 02f91d5 commit eefcc1b
Show file tree
Hide file tree
Showing 1,028 changed files with 47,537 additions and 46,178 deletions.
1 change: 1 addition & 0 deletions .github/scripts/cleanspell.pl
Expand Up @@ -71,6 +71,7 @@
$_ =~ s/curl_url_(dup)//g;
$_ =~ s/curl_pushheader_by(name|num)//g;
$_ =~ s/libcurl-(env|ws)//g;
$_ =~ s/libcurl\\-(env|ws)//g;
$_ =~ s/(^|\W)((tftp|https|http|ftp):\/\/[a-z0-9\-._~%:\/?\#\[\]\@!\$&'()*+,;=]+)//gi;
print O $_;
}
Expand Down
13 changes: 8 additions & 5 deletions .github/scripts/spellcheck.words
Expand Up @@ -83,6 +83,7 @@ CCC
CDN
CentOS
CFLAGS
cflags
CGI's
CHACHA
chacha
Expand Down Expand Up @@ -138,10 +139,11 @@ cshrc
CTRL
cURL
CURLcode
curldown
CURLE
CURLH
CURLINFO
curlimages
CURLINFO
curlrc
curltest
customizable
Expand All @@ -161,8 +163,8 @@ deepcode
DELE
DER
deselectable
Deserialized
deserialization
Deserialized
destructor
detections
dev
Expand Down Expand Up @@ -350,11 +352,10 @@ interoperable
interoperates
IoT
ipadOS
IPCXN
IPFS
IPNS
ipld
trustless
IPCXN
IPNS
IPv
IPv4
IPv4/6
Expand Down Expand Up @@ -804,6 +805,7 @@ TPF
TrackMemory
transcode
Tru
trustless
Tse
Tsujikawa
TTL
Expand All @@ -826,6 +828,7 @@ unencoded
unencrypted
unescape
Unglobbed
Unicode
UNICOS
unix
UnixSockets
Expand Down
10 changes: 9 additions & 1 deletion .github/scripts/verify-examples.pl
Expand Up @@ -91,12 +91,20 @@ sub extract {
return ($fail ? 0 : $l);
}

my $count;
for my $m (@files) {
print "Verify $m\n";
#print "Verify $m\n";
my $out = extract($m);
if($out) {
$error |= testcompile($m);
$error |= checksrc($m);
}
$count++;
}
if(!$error) {
print "Verified $count man pages ok\n";
}
else {
print "Detected problems\n";
}
exit $error;
3 changes: 3 additions & 0 deletions .github/workflows/man-examples.yml
Expand Up @@ -28,5 +28,8 @@ jobs:
- name: Checkout
uses: actions/checkout@v4

- name: render nroff versions
run: autoreconf -fi && ./configure --without-ssl --without-libpsl && make -C docs

- name: verify examples
run: ./.github/scripts/verify-examples.pl docs/libcurl/curl*.3 docs/libcurl/opts/*.3
3 changes: 2 additions & 1 deletion .github/workflows/proselint.yml
Expand Up @@ -42,7 +42,8 @@ jobs:
"checks": {
"typography.diacritical_marks": false,
"typography.symbols": false,
"annotations.misc": false
"annotations.misc": false,
"security.password": false
}
}
JSON
Expand Down
6 changes: 0 additions & 6 deletions Makefile.am
Expand Up @@ -153,12 +153,6 @@ dist-hook:
cp -p $$file $(distdir)$$strip; \
done)

html:
cd docs && $(MAKE) html

pdf:
cd docs && $(MAKE) pdf

check: test examples check-docs

if CROSSCOMPILING
Expand Down
16 changes: 8 additions & 8 deletions appveyor.yml
Expand Up @@ -84,7 +84,7 @@ environment:
ENABLE_UNICODE: 'OFF'
HTTP_ONLY: 'OFF'
TESTING: 'ON'
DISABLED_TESTS: '!1139 !1501'
DISABLED_TESTS: '!1139 !1501 !1140 !1173 !1177 !1477'
- job_name: 'CMake, VS2022, Debug, x64, Schannel, Static, Unicode'
APPVEYOR_BUILD_WORKER_IMAGE: 'Visual Studio 2022'
BUILD_SYSTEM: CMake
Expand All @@ -95,7 +95,7 @@ environment:
ENABLE_UNICODE: 'ON'
HTTP_ONLY: 'OFF'
TESTING: 'ON'
DISABLED_TESTS: '!1139 !1501'
DISABLED_TESTS: '!1139 !1501 !1140 !1173 !1177 !1477'
- job_name: 'CMake, VS2022, Debug, x64, no SSL, Static'
APPVEYOR_BUILD_WORKER_IMAGE: 'Visual Studio 2022'
BUILD_SYSTEM: CMake
Expand All @@ -106,7 +106,7 @@ environment:
ENABLE_UNICODE: 'OFF'
HTTP_ONLY: 'OFF'
TESTING: 'ON'
DISABLED_TESTS: '!1139 !1501'
DISABLED_TESTS: '!1139 !1501 !1140 !1173 !1177 !1477'
- job_name: 'CMake, VS2022, Debug, x64, no SSL, Static, HTTP only'
APPVEYOR_BUILD_WORKER_IMAGE: 'Visual Studio 2022'
BUILD_SYSTEM: CMake
Expand All @@ -117,7 +117,7 @@ environment:
ENABLE_UNICODE: 'OFF'
HTTP_ONLY: 'ON'
TESTING: 'ON'
DISABLED_TESTS: '!1139 !1501'
DISABLED_TESTS: '!1139 !1501 !1140 !1173 !1177 !1477'
# generated CMake-based MSYS Makefiles builds (mingw cross-compiling)
- job_name: 'CMake, mingw-w64, gcc 13, Debug, x64, Schannel, Static, Unicode, Unity'
APPVEYOR_BUILD_WORKER_IMAGE: 'Visual Studio 2022'
Expand All @@ -128,7 +128,7 @@ environment:
ENABLE_UNICODE: 'ON'
HTTP_ONLY: 'OFF'
TESTING: 'ON'
DISABLED_TESTS: '!1086 !1139 !1451 !1501'
DISABLED_TESTS: '!1086 !1139 !1451 !1501 !1140 !1173 !1177 !1477'
ADD_PATH: 'C:/msys64/mingw64/bin'
MSYS2_ARG_CONV_EXCL: '/*'
BUILD_OPT: -k
Expand All @@ -142,7 +142,7 @@ environment:
ENABLE_UNICODE: 'ON'
HTTP_ONLY: 'OFF'
TESTING: 'ON'
DISABLED_TESTS: '!1086 !1139 !1451 !1501'
DISABLED_TESTS: '!1086 !1139 !1451 !1501 !1140 !1173 !1177 !1477'
ADD_PATH: 'C:/mingw-w64/x86_64-7.2.0-posix-seh-rt_v5-rev1/mingw64/bin'
MSYS2_ARG_CONV_EXCL: '/*'
BUILD_OPT: -k
Expand All @@ -156,7 +156,7 @@ environment:
HTTP_ONLY: 'OFF'
TESTING: 'ON'
# test 286 disabled due to https://github.com/curl/curl/issues/12040
DISABLED_TESTS: '~286 !1086 !1139 !1451 !1501'
DISABLED_TESTS: '~286 !1086 !1139 !1451 !1501 !1140 !1173 !1177 !1477'
ADD_PATH: 'C:/msys64/mingw64/bin'
MSYS2_ARG_CONV_EXCL: '/*'
BUILD_OPT: -k
Expand All @@ -170,7 +170,7 @@ environment:
ENABLE_UNICODE: 'OFF'
HTTP_ONLY: 'OFF'
TESTING: 'ON'
DISABLED_TESTS: '!1086 !1139 !1451 !1501'
DISABLED_TESTS: '!1086 !1139 !1451 !1501 !1140 !1173 !1177 !1477'
ADD_PATH: 'C:/mingw-w64/i686-6.3.0-posix-dwarf-rt_v5-rev1/mingw32/bin'
MSYS2_ARG_CONV_EXCL: '/*'
BUILD_OPT: -k
Expand Down
6 changes: 2 additions & 4 deletions docs/.gitignore
Expand Up @@ -2,7 +2,5 @@
#
# SPDX-License-Identifier: curl

*.html
*.pdf
curl.1
*.1.dist
*.1
*.3
2 changes: 1 addition & 1 deletion docs/CONTRIBUTE.md
Expand Up @@ -116,7 +116,7 @@ features are working as they are supposed to. To maintain this situation and
improve it, all new features and functions that are added need to be tested
in the test suite. Every feature that is added should get at least one valid
test case that verifies that it works as documented. If every submitter also
posts a few test cases, it will not end up as a heavy burden on a single person!
posts a few test cases, it will not end up as a heavy burden on a single person.

If you do not have test cases or perhaps you have done something that is hard
to write tests for, do explain exactly how you have otherwise tested and
Expand Down
125 changes: 125 additions & 0 deletions docs/CURLDOWN.md
@@ -0,0 +1,125 @@
# curldown

A markdown-like syntax for libcurl man pages.

## Purpose

A text format for writing libcurl documentation in the shape of man pages.

Make it easier for users to contribute and write documentation. A format that
is easier on the eye in its source format.

Make it harder to do syntactical mistakes.

Use a format that allows creating man pages that end up looking exactly like
the man pages did when we wrote them in nroff format.

Take advantage of the fact that people these days are accustomed to markdown
by using a markdown-like syntax.

This allows us to fix issues in the nroff format easier since now we generate
them. For example: escaping minus to prevent them from being turned into
Unicode by man.

Generate nroff output that looks (next to) *identical* to the previous files,
so that the look, existing test cases, HTML conversions, existing
infrastructure etc remain mostly intact.

Contains meta-data in a structured way to allow better output (for example the
see also information) and general awareness of what the file is about.

## File extension

Since curldown looks similar to markdown, we use `.md` extensions on the
files.

## Conversion

Convert **from curldown to nroff** with `cd2nroff`. Generates nroff man pages.

Convert **from nroff to curldown** with `nroff2cd`. This is only meant to be
used for the initial conversion to curldown and should ideally never be needed
again.

Convert, check or clean up an existing curldown to nicer, better, cleaner
curldown with **cd2cd**.

Mass-convert all curldown files to nroff in specified directories with
`cdall`:

cdall [dir1] [dir2] [dir3] ..

## Known issues

The `cd2nroff` tool does not yet handle *italics* or **bold** where the start
and the end markers are used on separate lines.

The `nroff2cd` tool generates code style quotes for all `.fi` sections since
the nroff format does not carry a distinction.

# Format

Each curldown starts with a header with meta-data:

---
c: Copyright (C) Daniel Stenberg, <daniel.se>, et al.
SPDX-License-Identifier: curl
Title: CURLOPT_AWS_SIGV4
Section: 3
Source: libcurl
See-also:
- CURLOPT_HEADEROPT (3)
- CURLOPT_HTTPAUTH (3)
---

All curldown files *must* have all the headers present and at least one
`See-also:` entry specified.

Following the header in the file, is the manual page using markdown-like
syntax:

~~~
# NAME
a page - this is a page descriving something
# SYNOPSIS
~~~c
#include <curl/curl.h>
CURLcode curl_easy_setopt(CURL *handle, CURLOPT_AWS_SIGV4, char *param);
~~~
~~~
Quoted source code should start with `~~~c` and end with `~~~` while regular
quotes can start with `~~~` or just be indented with 4 spaces.
Headers at top-level `#` get converted to `.SH`.
`nroff2cd` supports the `##` next level header which gets converted to `.IP`.
Write bold words or phrases within `**` like:
This is a **bold** word.
Write italics like:
This is *italics*.
Due to how man pages don't support backticks especially formatted, such
occurrences in the source will instead just use italics in the generated
output:
This `word` appears in italics.
When generating the nroff output, the tooling will remove superfluous newlines,
meaning they can be used freely in the source file to make the text more
readable.
All mentioned curl symbols that have their own man pages, like
`curl_easy_perform(3)` will automatically be rendered using italics in the
output without having to enclose it with asterisks. This helps ensuring that
they get converted to links properly later in the HTML version on the website,
as converted with `roffit`. This makes the curldown text easier to read even
when mentioning many curl symbols.
This auto-linking works for patterns matching `(lib|)curl[^ ]*(3)`.
2 changes: 1 addition & 1 deletion docs/GOVERNANCE.md
Expand Up @@ -179,4 +179,4 @@ this.
### Stop being a maintainer

If you (appear to) not be active in the project anymore, you may be removed as
a maintainer. Thank you for your service!
a maintainer. Thank you for your service.
2 changes: 1 addition & 1 deletion docs/HISTORY.md
Expand Up @@ -327,7 +327,7 @@ April: added the cyassl backend (later renamed to WolfSSL)

January: the curl tool defaults to HTTP/2 for HTTPS URLs

December: curl 7.52.0 introduced support for HTTPS-proxy!
December: curl 7.52.0 introduced support for HTTPS-proxy

First TLS 1.3 support

Expand Down

0 comments on commit eefcc1b

Please sign in to comment.