A high-performance command-line web crawler written in Go.
Crawl websites, extract hidden resources, respect or bypass robots policies, and discover application endpoints through a fast streaming architecture.
Webora is a lightweight, concurrent web crawler designed for developers, security researchers, and automation engineers. Instead of relying on browser automation or building large DOM trees in memory, Webora processes web content as streaming data, enabling fast crawling with a significantly smaller memory footprint.
It is capable of discovering URLs from HTML documents, JavaScript, CSS, forms, assets, comments, and sitemaps while providing extensive request customization through a modern CLI interface.
+----------------+
| CLI |
| Flag Parsing |
+-------+--------+
|
▼
Configuration Builder
|
▼
Request Configuration
(Headers, Cookies, UA, Proxy, TLS)
|
▼
Concurrent Worker Pool
|
▼
HTTP Request Engine
|
+----------------+----------------+
| |
▼ ▼
robots.txt Target Resources
| |
▼ ▼
Robots Parser HTML / JS / CSS Parser
| |
+---------------+-----------------+
▼
URL Normalization
▼
Duplicate Elimination
▼
Crawl Scheduler
▼
Stream to stdout
flowchart TD
A[User executes Webora] --> B[Parse CLI Flags]
B --> C[Configure HTTP Client]
C --> D[Download robots.txt]
D --> E{Robots Policy}
E -->|Ignore| F
E -->|Respect| F
E -->|Crawl Directives| G[Extract Sitemap URLs]
G --> F
F[Start Concurrent Crawl]
F --> H[Download Resource]
H --> I[Parse HTML]
I --> J[Extract URLs]
J --> K[Normalize Links]
K --> L[Remove Duplicates]
L --> M{Within Depth?}
M -->|Yes| H
M -->|No| N[Output Result]
Webora's request layer is designed to mimic real browser traffic while remaining lightweight.
flowchart LR
CLI --> Headers
Headers --> Cookies
Cookies --> UserAgent
UserAgent --> Proxy
Proxy --> TLS
TLS --> Delay
Delay --> HEAD
HEAD --> GET
GET --> Parser
Supported request capabilities include:
- Custom request headers
- Custom cookies
- User-Agent spoofing
- Proxy support
- Proxy authentication
- Configurable request delays
- Optional HEAD pre-flight optimization
- TLS verification control
- Request timeout configuration
Instead of constructing complete DOM trees, Webora parses documents as continuous streams.
flowchart LR
HTML --> HTMLParser
JavaScript --> JSLexer
CSS --> CSSLexer
HTMLParser --> URLExtraction
JSLexer --> EndpointExtraction
CSSLexer --> AssetExtraction
URLExtraction --> Normalizer
EndpointExtraction --> Normalizer
AssetExtraction --> Normalizer
Normalizer --> CrawlQueue
This streaming architecture allows Webora to:
- Process pages immediately as data arrives.
- Reduce memory allocations.
- Handle large documents efficiently.
- Discover endpoints hidden inside JavaScript and CSS.
- Maintain high crawl throughput.
Large-scale crawling can consume significant amounts of memory when millions of URLs are discovered.
Webora minimizes resource consumption through several optimizations.
flowchart TD
A["Long URL"]
B["FNV-1a Hash"]
C["64-bit Integer"]
D["Hash Set"]
E["Duplicate Detection"]
A --> B
B --> C
C --> D
D --> E
- Streaming HTML parsing without DOM construction
- FNV-1a hashing for compact URL storage
- Constant-time duplicate detection
- MIME-type filtering before parsing
- File-extension pre-screening
- Configurable worker pool
- Configurable crawl depth
flowchart LR
A["robots.txt"]
B["Streaming Parser"]
C["Disallow Rules"]
D["Hash Set"]
E["URL Validation"]
F["Crawler"]
A --> B
B --> C
C --> D
D --> E
E --> F
Depending on configuration, Webora can:
- Ignore robots rules
- Respect robots directives
- Extract sitemap declarations
- Crawl directive URLs for additional discovery
Webora discovers URLs from multiple sources during a crawl.
| Source | Description |
|---|---|
| HTML | Anchors, images, forms, scripts, stylesheets and media resources |
| JavaScript | API endpoints, fetch requests and embedded URLs |
| CSS | url() properties and imported assets |
| HTML Comments | Optional brute-force discovery of forgotten endpoints |
| robots.txt | Disallow rules and sitemap declarations |
webora [flags] <url>webora https://example.comwebora -depth -1 https://example.comwebora -js https://example.comwebora -include "/blog,/docs" https://example.comwebora -match-regex "\/youtube\/[0-9]+[a-zA-Z0-9\-]+$" https://example.comwebora -brute https://example.comwebora -robots respect https://example.comwebora \
-proxy-auth user:password \
-user-agent "Mozilla/5.0" \
https://example.com| Flag | Description |
|---|---|
-depth |
Maximum crawl depth |
-workers |
Concurrent workers |
-delay |
Delay between requests |
-timeout |
HTTP timeout |
-header |
Custom request headers |
-cookie |
Custom cookies |
-user-agent |
User-Agent override |
-robots |
robots.txt policy |
-proxy-auth |
Proxy authentication |
-skip-ssl |
Disable certificate verification |
-subdomains |
Include subdomains |
-headless |
Skip HEAD pre-flight requests |
-js |
Parse JavaScript |
-css |
Parse CSS |
-brute |
Scan HTML comments |
-include |
Include specific URLs |
-match-regex |
Match URLs using a regex |
-jsonl |
Output results as JSON Lines |
-status |
Report HTTP status codes |
-jitter |
Add random jitter duration |
-proxies |
Rotates traffic through a pool of proxies |
-custom-agents |
Rotates traffic through a pool of custom User-Agent headers |
✔ Concurrent worker pool
✔ Streaming SAX HTML parser
✔ JavaScript lexical parsing
✔ CSS lexical parsing
✔ Constant-time duplicate detection
✔ Minimal memory allocations
✔ Configurable crawl depth
✔ Optimized request pipeline
✔ MIME-aware resource filtering
✔ Cross-platform CLI
Webora is designed to be useful across development, security, DevOps, and automation workflows.
Discover REST, GraphQL, and hidden application endpoints referenced inside HTML, JavaScript, and CSS files.
Useful for:
- API documentation generation
- Reverse engineering web applications
- Security assessments
- Integration analysis
Quickly enumerate publicly accessible resources to understand an application's exposed surface.
Discover:
- Login portals
- Admin panels
- API routes
- Static assets
- Documentation pages
- Hidden directories
Collect URLs before running vulnerability scanners or manual penetration tests.
Webora can help identify:
- Forgotten endpoints
- Development paths
- Internal API references
- Backup files
- Archived resources
- Comment-based URLs
Efficiently crawl websites containing thousands or millions of pages while maintaining a low memory footprint through streaming parsing and concurrent execution.
Ideal for:
- Enterprise websites
- Documentation platforms
- E-commerce stores
- News portals
Automatically build an inventory of application endpoints referenced throughout a website.
Useful for:
- API governance
- Service inventory
- Migration projects
- Documentation validation
Crawl websites and collect internal links for validation or integration with external link-checking tools.
Useful during:
- Website migrations
- SEO audits
- Content management
- Documentation maintenance
Identify resources referenced across a website including:
- JavaScript bundles
- Stylesheets
- Images
- Fonts
- Media files
- Downloadable documents
Useful for optimization and auditing.
Extract information from robots.txt and sitemap declarations to discover pages that may not be linked through normal navigation.
Useful for:
- Site auditing
- SEO analysis
- Reconnaissance
- Content discovery
Run Webora as part of scheduled automation to detect newly exposed endpoints, pages, or assets over time.
Useful for:
- CI/CD pipelines
- Security monitoring
- Website change detection
- Compliance checks
Use JSON Lines output to integrate Webora into automation workflows.
Example integrations include:
- SIEM platforms
- Search indexing
- Data lakes
- Analytics pipelines
- Asset inventories
git clone https://github.com/HadeedTariq/webora
cd webora
go build -o webora ./cmd/weborawebora \
-depth 3 \
-workers 16 \
-delay 150ms \
-robots respect \
-js \
-css \
https://example.comWebora is built around a simple principle:
Process data as streams, minimize memory usage, maximize crawl throughput, and provide a flexible command-line interface that scales from small websites to large web applications.
Rather than depending on heavyweight browser automation, Webora performs efficient HTTP crawling using optimized parsers, configurable networking, and concurrent execution to achieve fast and predictable performance.