Some elements (headings, code blocks and callouts) are missing on the MDN Web Docs

[RyotaUshio]

Hi, thank you for the amazing projects! I noticed the Web Clipper misses all or some lines in some code blocks on the MDN Web Docs. Also, some callouts are also missing.

For example: https://developer.mozilla.org/en-US/docs/Web/HTTP/Guides/CORS

In the "Simple requests" section

• The second code block that starts with GET /resources/public-data/ HTTP/1.1: All HTTP headers (line 2-) are missing.

Original:

GET /resources/public-data/ HTTP/1.1
Host: bar.other
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:71.0) Gecko/20100101 Firefox/71.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us,en;q=0.5
Accept-Encoding: gzip,deflate
Connection: keep-alive
Origin: https://foo.example

Clipped:

```
GET /resources/public-data/ HTTP/1.1
```

• The next HTTP code block that starts with HTTP/1.1 200 OK: All HTTP headers (line 2-8) are missing.

Original:

HTTP/1.1 200 OK
Date: Mon, 01 Dec 2008 00:23:53 GMT
Server: Apache/2
Access-Control-Allow-Origin: *
Keep-Alive: timeout=2, max=100
Connection: Keep-Alive
Transfer-Encoding: chunked
Content-Type: application/xml

[…XML Data…]

Clipped:

```
HTTP/1.1 200 OK

[…XML Data…]
```

• The next code block is completely missing.

Original:

Access-Control-Allow-Origin: *

Clipped:

• The next code block is completely missing.

Access-Control-Allow-Origin: https://foo.example

Clipped:

• The callout (starting with "Note: When responding to ...") at the end of that section is also missing.

In the "Preflighted requests" section

• The first JS code block is completely missing.
• The callout below the first image in the section (starting with "Note: As described below, ...") is missing.
• The next code block (HTTP; starting with OPTIONS /doc HTTP/1.1) is completely missing.
• The next code block (HTTP; starting with Access-Control-Request-Method: POST) is completely missing.
• The next code block (HTTP; starting with Access-Control-Allow-Origin: https://foo.example) is completely missing.
• The next code block (HTTP; starting with POST /doc HTTP/1.1) is missing all HTTP headers (line 2-14, 19-27).

... and more.

[RyotaUshio]

I found another example where a heading is ignored.
I tried clipping the following page with Web Clipper 0.11.9, but the heading "Two XSS examples" did not appear in the result.

https://developer.mozilla.org/en-US/docs/Web/Security/Attacks/XSS

[mdbraber]

@RyotaUshio

the first examples are because of the string 'header' matching in the text, after Defuddle has determined it's likely not content.

defuddle/src/scoring.ts

Line 31 in 34e540e

'header',

the last example is because of this line matching:

defuddle/src/constants.ts

Line 178 in 34e540e

'a[href^="#"][class*="anchor" i]',

A suggestion for a solution would be to add 'code-example' to the list of allowed contentIndicators here

defuddle/src/scoring.ts

Line 3 in 34e540e

const contentIndicators = [

@kepano what would your advice be in fixing these types of exceptions? In constants.ts a list of these types of constants is kept, also with references to e.g. why some exceptions are made. I could see that the constants from scoring.ts also are kept in constants.ts and are ultimately amendable in specific situations (see FR #100)

[kepano]

Should be fixed in 0.11.0