Welcome to SPIDER, the ultimate tool for enhancing your JavaScript website's visibility to bots and crawlers. In today's web landscape, where JavaScript-driven sites are prevalent, it's crucial to ensure that your content is effectively indexed and accessible. SPIDER is here to help. By seamlessly interpreting and presenting your website's content to bots, SPIDER boosts your online presence like never before.
For more details, check out our Knowledge Base.
This is the SPIDER middleware configuration for an nginx server to allow Google (and other crawlers) to crawl your javascript website.
This file is based on this and it was updated following the recommendations in the comments
The nginx.conf in this repository is full file for reference.
Nginx config for
- Single Page Application: nginx.conf
- PHP application: nginx-php/nginx.conf
- reverse proxy: nginx-reverse-proxy/nginx.conf
To debug the page sent to utility-server.com use the following log format:
log_format spider_debug '[$time_local] $remote_addr - $remote_user - $server_name user-agent: $http_user_agent spider: $spider: $request to: $upstream_addr upstream_response_time: $upstream_response_time msec $msec request_time $request_time';