- Improve "Scanning Azure services" UI
- Make sure each domain gets:
- DNS resolution
- DNS inspection
- Service Scans
- Improve constants and general response object property names
- WTF is previousAWS?
- Improve description & synopsis for v2.0.0 service detection capabilities
-
Fix handling services which resolve to known DNS apex zones but are also AWS services (the AWS service details should enrich the DNS zone apex details rather than replace them)
-
Fix AWS service lookup where the IP address appears in multiple CIDR blocks
-
If the input does not have a valid top level domain then assume it's a file, and report
file not found
accordingly; e.g.ccc filename.txt
as.txt
is not a valid TLD -
A URL whose domain is not resolved (
ENOTFOUND
) is still included in the CDN Detection and is reported asUnknown
- e.g.node .\ccc.js iplayer.bbc.co.uk
- Need to handle an empty
answer.answer[]
array, and parseAnswer() returning something more useful than'no_address'
- Need to handle an empty
-
If a request redirects to a new domain and we follow that redirect, the target domain isn't in the
uniqueDomains
Set -
Handle misspelt filename being treated as URL
-
--interval
&--iterations
do not work as intended -
Check for invalid hostnames in valid URLs (where
https://*.allowed.com/
passes the valid-URL test but*.allowed.com
is validated as a domain):Checking if [https://*.allowed.com/] is a file, URL or bare domain ... It's a valid URL
-
Fix the progress indicator flickering. Perhaps move to an alternative library like Node.CLI-Progress
-
Investigate the wisdom of waiting for the external app to close before continuing when opening the
.csv
file. Perhaps make the behavior a switch:(async () => { // Opens the image in the default image viewer and waits for the opened app to quit. await open(filename); })();
-
--help
screen doesn't reflect all command line switches- export
- open
- http options
- debug
- Add
mktoweb.com
domain detection => Adobe Marketo Engage - Add ability to extract URLs from the base HTML returned, and add them to the URLs array - perhaps using extract-urls
- Move DNS timeout (5000 - ccc-dnc.js:150) into configuration and give it a command line override.
- Add Azure IP ranges
json
toservice.providers
. - Add CloudFlare IP ranges to
service.providers
. - Add Fastly IP ranges to
service.providers
. - Add GEO location lookup of IP address hosting each resource via IP Who Is
- Add option to emulate
user-agent
of popular browsers or provide a custom user-agent string. - Add an asynchronous update of AWS ip-ranges.json based on the
syncToken
property. - Add DNS options to
configuration.json
.- Allow command line override for DNS options.
- Add a mechanism to provide advice - when a
read_timeout
occurs inform the user of the argument to increase it. - Improve CDN Detection by examining telltale response headers (such as
server: cloudfront
). - Detect if
--list-header-collections
is being piped to a file and output rawjson
(i.e. only useprettyjson
for console output). - Add colour indicators for the response headers:
-
vary
- where*
,user-agent
,cookie
are all anti-patterns for CDN caching. -
content-encoding
- where not beinggzip
,br
, et al is sub-optimal.
-
- HTTP/2 Support (or reporting support against each unique domain).
- Change the
exportToCSV()
function to save files to accc
specific subfolder. - Add a modifier to
--open
which opens the folder. - Add support for input file type Lighthouse
.json
to extract resource URLs from. - Allow modification of DNS question (
resolver
etc) and move defaults to config file. - Report on CNAME TTL for each unique domain.
- eTag support - Allow conditional
GET
requests such asIf-None-Match
to analyse Entity Tags. - Allow request headers to be injected.
- Automatically inject request headers array as part of header collection.
- i.e. Add
fastly-debug:1
to all requests when the Fastly header collection is used.
- i.e. Add
- Automatically inject request headers array as part of header collection.
- Add console width warning if it's too narrow (
process.stdout.columns
). - Some sites (such as
www.etsy.com
&www.amazon.co.uk
) use multiple CDNs for the domain depending where the client is. Mention this in a README section, but fix it by turningddig
into a library and querying multiple resolvers for the fullCNAME
chain before performing CDN detection. - Add a reason to
cdnDetection()
output when--verbose
is enabled. - Command line arguments for
needle
http options:-
timeout
-
response_timeout
-
read_timeout
-
follow
-
compress
- et al
-
- Implement full debug request/response logging:
- Perhaps to a
.har
file. - Export all response headers to separate
csv
file whendebug
||verbose
is enabled.
- Perhaps to a
- Provide a friendly interpretation of caching based on CDN's x-cache documentation.
- Perhaps workout a score based upon
x-cache
,cache-control
,eTag
, et al.
- Perhaps workout a score based upon
- Investigate merits of implementing a custom http_agent
- Default DNS resolver is hardcode to
8.8.8.8
but should be local DNS resolverdns.getServers()
- The customised
user-agent
isn't being set properly inneedle
client request - Add AWS service detection based upon AWS IP Ranges and using cidr.contains(address)
-
The status column may displayundefined
when the error occurred at the network level (not a HTTP response) -
Add support for input file type.har
to extract resource URLs from it -
--open
doesn't work as intended (it doesn't open the csv file) -
--export
doesn't work as intended (it doesn't turn exportToCSV on or off) -
--headers [collection]
is case sensitive meaningccc https://www.mozilla.org/ --headers cors
causes a warning whileccc https://www.mozilla.org/ --headers CORS
succeeds -
--help
examples should matchreadme
examples -
getDefaults()
appears to called three times when executingccc --list-header-collections
-
--list-header-collections
is not sorted alphabetically -
--list-response-headers
also performs CDN detection but should probably be constrained to just listing the headers -
Add padding character to CDN Detection table to aid readability -
Display an activity indicator whilst the HTTP requests are being made (maybe ora) -
Add an indicator (®) to the response output row when a redirect was followed, and add the redirect count to the rawcsv
export file -
Add the actualhostname
being resolved toparseAnswer()
so that it's included in thedetermineCDN()
logic -
Export to CSV -
Expandsettings
to incorporateneedle
's options json -
Filter response headers based on selected header collection -
Format output into columns -
Implement--header-collections
-
Implement--headers
-
Collect all unique response headers and optionally list them out--list-response-headers
-
Add timestamp into output columns -
Colour output to indicate cache hit & cache miss -
Implement iterations -
Expanduser-agent
{variables} when reading config -
Investigate the spurious400
responses -
Allow customisation of user-agent string (avoid bot detection) -
Investigate why req.path hasneedle
optionsjson
concatenated -
AddedexportToCSV
to settings -
Add option toopenAfterExport
-
Implement command line switch to enable/disableopenAfterExport
-
Implement command line switch to enable/disableexportToCSV
-
Implementsettings.options.httpOptions
passed through toneedle
-
Attempt CDN detection based onCNAME
apex domain. -
ReplaceCDN detection
withDNS Inspection
-
CDN detection complete on {total} unique domains
isn't an accurate message. DNS inspection complete, perhaps