Change in 404 error reporting?

[humantex]

This may be a very minor issue, unless it's a result of some change in Google's own code and then it's more of a non-Koken issue as far as their report goes, but....

Google's crawl report for my site (in their webmaster search console) shows a single consistent error starting on 2/10/2017 for 'error/404/' as a "page not found". I understand 404's, so it's not that the error itself is entirely incorrect - but it hasn't shown up as an error in any Google reports between last November on up to February 10th, and the bot regularly crawls the site according to my reports and site logs. I've made no template changes to cause the issue.

After some testing I found that my 404 page generally works as expected on errors, and it also displays the results for routed_variables.code as an error code from the template. The one exception where it won't render correctly is when I manually type in /error after my domain name as a directory/folder route, I see a direct rendering of the theme's 'error.lens' template with a blank error code. It's still looks complete, with header and footer templates - otherwise it's what you'd visually expect from a non-existent resource producing a 404 error. The only problem seems to be that there's no proper redirect or a display of Koken's internal 404 error code.

i.e., https://lumiworx.com/error/ will show error.lens as-is, versus something like https://lumiworx.com/jo4deswqut that will properly redirect to /error/404/ and show the 404 error code.

This all may be trivial and not worth investigating - and/or unrelated - but experience tells me that when Google's reports start showing errors there may be something else in the works.

[humantex]

I think this might be more than what it first appeared to be - or - I haven't noticed this particular behavior until Google pointed it out.

If you append some of the lens template names to a site's domain URL, the site will render the template as-is, without any content - or unexpected content - and without properly redirecting the browser to a 404 page or some other appropriate resource. The theme in use doesn't seem to have any bearing on the behavior - only on which templates are susceptible, based on what templates it includes. This same behavior affects the premium themes I've tried from the demo site and some from the 'examples' user sites.

Affected lens templates - in my case, while using a modified Elementary theme:

album.lens - ( i.e.: https://lumiworx.com/album ) Shows only the header and breadcrumbs, with no content and no footer.
date.lens - shows only the contents with one of the <aside> columns; presumably the 'timeline_aside.html' layout.
error.lens - As noted, above.
lightbox.lens - shows an all-black lightbox with a spinner that never stops rotating.
page.lens - Only blank main content, with site header and no footer.
tag.lens - Only blank main content, with site header and no footer.
essay.lens - Shows only the site header and 'mcol' <article> header, with share links - no main content or footer.
set.lens - Shows only the site header and breadcrumbs, with no content and no footer.

There might be others too, but I only checked these names specifically. I can also say with no hesitation, that I've made no changes to any core files or modified the theme with any path or alias busting code of my own. I made one change to the root htaccess file to force a redirect for any page requests to be served via SSL, but left all other code and regex sections untouched. And lastly, there are no unofficial plugins or other 3rd party add-ins in use on the site.

All of this may be well known and expected behavior, but if that is the case, I think it might be better to have redirects in place for any/all lens and layout templates in the future. I'd hope that a user gets guided to real resources and not end up on broken pages - even by accident.

[humantex]

Well, Google's webmaster tool panel reports has been cycling through crawls on several of the page paths as noted above - and posting them as consistent errors. As a temporary fix, I've modified the site's htaccess file to do some permanent redirects. There is one which will not work. Trying to redirect '/error' to '/error/404' results in a never-ending loop in Firefox (v52), so it's been commented out for now.

I'm far from being a wiz when it comes to coding Apache rules or doing regex stuff, so this will have to do for the time being. I don't currently have any sets associated within my albums, so both '/set' and '/sets' are getting redirected back to root.

I have absolutely no idea why Google is looking for these paths/pages, as they are certainly not in my sitemap file - which, according to the sitemap reports, really IS being read and used for directing bot crawls.

Redirect 301 /album /albums
Redirect 301 /essay /essays
Redirect 301 /date /timeline
Redirect 301 /set /
Redirect 301 /sets /
# Redirect 301 /error /error/404
Redirect 301 /lightbox /
Redirect 301 /page /

[BlackSkorpio]

Not sure, bit this is mostly a theme issue...
And should be addressed by any theme dev, even if it would be cool if Koken would show the koken.note publicly in album, essay, date, set & page.lens. Just to do our job easier ;)

OxyGen v3.3.1 that will be released the coming days will address this issue:, and have a solid fix theme whise.

[humantex]

Sorry, but I doubt this is theme related when it's basic Koken-created core file coding that handles all the page and path aliasing via it's own php and script calls along with the Koken-provided .htaccess rewrite rules. Every default Koken install using the free templates would report these same errors. I can only assume that the paid themes would exhibit the same results. I have made no changes to Koken's core files, nor have I altered the .htaccess file in any way that would account for mangling any pathing data as Koken would usually direct it.

My theme is based on the free Elementary theme and all I've added to it or modified is a few official Koken Lens Tags (for tag search, 'order by' in <koken:loop> calls, and extending the EXIF/IPTC data display), standard HTML/CSS styling for content, and a few scripts that are strictly for presentation only. Nothing has been added or changed that could even attempt to modify URL paths. I have added no new .lens files/templates other than those original to the theme, so I haven't introduced any new variables to be misinterpreted.

For this particular site there's no need to provide any functionality beyond what is already included within Koken's official site/plug-in releases, and I won't add anything theme-wise to modify something that isn't presentation related. That's what plug-ins and core files are meant to cover, and I won't jeopardize a later clash with core code to fix site behavior issues just to end up with a broken site. And... what about other users who don't use a paid or custom theme or can't/won't do their own paid/custom theme or site coding? They should benefit from any fix by doing a standard site and theme update.

My entire .htaccess file [below] shows my alternate code at the top that forces SSL site-wide (which doesn't alter any paths whatsoever - only the protocol to use), then the newly added 301 redirects to handle the Google issue. The remainder of the file from line # 25 on [i.e., # DO NOT EDIT BELOW THIS LINE #] is all of Koken's original coding that obviously does alter the site's paths for navigation and creating pretty/clean URL's, all the page/image caching, image resolution display redirects, and more.

If there is anything that could be included in ANY theme to fix a pathing issue with a theme's use of .lens templates and/or layout files, then it should be included with any Koken-supplied theme as the official patch. If there's something I can add into my theme in the mean time, I'd be happy to insert the approved code while that fix is in the works for the standard themes. If core code needs changing - including anything below line 25 in htaccess, as in my case - then I'll wait until Koken publishes it's official change before editing or updating.

The site's current .htaccess file:

## Uncomment the following block to force SSL when accessing /admin
# <IfModule mod_rewrite.c>
#	RewriteEngine On
# 	RewriteCond %{REQUEST_URI} /admin/
# 	RewriteCond %{SERVER_PORT} 80
# 	RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
# </IfModule>

# Forces SSL site-wide, and not just when accessing  /admin
<IfModule mod_rewrite.c>
	RewriteEngine On
	RewriteCond %{HTTPS} !=on
	RewriteRule ^ https://%{HTTP_HOST}%{REQUEST_URI} [L,R=301]
</IfModule>

Redirect 301 /album /albums
Redirect 301 /essay /essays
Redirect 301 /date /timeline
Redirect 301 /set /
Redirect 301 /sets /
# Redirect 301 /error /error/404
Redirect 301 /lightbox /
Redirect 301 /page /

####### DO NOT EDIT BELOW THIS LINE ########

#MARK#######################################
########	KOKEN .htaccess rules	########
############################################

## Make sure settings.css.lens is rendered as CSS
AddType text/css .lens

## SVG MIME type as some servers don't have it
AddType image/svg+xml .svg

## UTF-8 encoding for everything
AddDefaultCharset utf-8

DirectoryIndex index.php index.html

## Enable gzip.
## Highly recommended as it will increase speed for
## both the console and your published site.
<IfModule mod_deflate.c>
	AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/json application/javascript application/x-javascript
</IfModule>
## END gzip

## Rewrite Rules (Pretty URLs)
## These rules remove index.php/ from your published site links
## and also speed up the serving of cached images.
<IfModule mod_rewrite.c>
	RewriteEngine On
	RewriteBase /

	# Pass images requests back to PHP if they do not exist
	RewriteCond %{REQUEST_METHOD} =GET
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteCond %{REQUEST_URI} /storage/cache/images(/(([0-9]{3}/[0-9]{3})|custom)/.*)$
	RewriteRule . /i.php?path=%1 [L]

	# Pass albums requests back to PHP if they do not exist
	RewriteCond %{REQUEST_METHOD} =GET
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteCond %{REQUEST_URI} /storage/cache/albums(/([0-9]{3}/[0-9]{3})/.*)$
	RewriteRule . a.php?path=%1 [L]

	# Catch root requests (pjax)
	RewriteCond %{REQUEST_METHOD} =GET
	RewriteCond %{REQUEST_URI} ^/?$
	RewriteCond %{QUERY_STRING} _pjax=
	RewriteCond %{DOCUMENT_ROOT}/storage/cache/site/index/cache.phtml -f
	RewriteRule .* /storage/cache/site/index/cache.phtml [L]

	# Catch root requests
	RewriteCond %{REQUEST_METHOD} =GET
	RewriteCond %{REQUEST_URI} ^/?$
	RewriteCond %{QUERY_STRING} !_pjax=
	RewriteCond %{DOCUMENT_ROOT}/storage/cache/site/index/cache.html -f
	RewriteRule .* /storage/cache/site/index/cache.html [L]

	# Catch site requests (pjax)
	RewriteCond %{REQUEST_METHOD} =GET
	RewriteCond %{QUERY_STRING} _pjax=
	RewriteCond %{DOCUMENT_ROOT}/storage/cache/site%{REQUEST_URI}cache.phtml -f
	RewriteRule . /storage/cache/site%{REQUEST_URI}cache.phtml [L]

	# Catch site requests
	RewriteCond %{REQUEST_METHOD} =GET
	RewriteCond %{QUERY_STRING} !_pjax=
	RewriteCond %{HTTP_COOKIE} !share_to_tumblr=
	RewriteCond %{DOCUMENT_ROOT}/storage/cache/site%{REQUEST_URI}cache.html -f
	RewriteRule . /storage/cache/site%{REQUEST_URI}cache.html [L]

	# CSS / RSS caching
	RewriteCond %{REQUEST_METHOD} =GET
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteCond %{DOCUMENT_ROOT}/storage/cache/site%{REQUEST_URI} -f
	RewriteRule . /storage/cache/site%{REQUEST_URI} [L]

	# Rewrite any old URLs that still use index.php?/this/that syntax
	RewriteCond %{QUERY_STRING} ^/(.*)
	RewriteRule ^index.php %1? [R,L]

	# Rewrite any old URLs that still use index.php/this/that syntax
	RewriteRule ^index.php/(.*)$ $1 [R,L]

	# Catch root requests
	RewriteRule ^(index.php)?$ /app/site/site.php?url=/ [L,QSA]

	# Do not enable path rewriting for files that exist
	RewriteCond %{REQUEST_FILENAME} !-f
	RewriteCond %{REQUEST_FILENAME} !-d
	RewriteCond	%{REQUEST_FILENAME} !favicon.ico

	# For requests that are not actual files
	# rewrite to index.php?/PATH
	RewriteRule ^(.*)$ /app/site/site.php?url=/$1 [L,QSA]
</IfModule>

## This ruleset ensures core Koken JS and CSS are cached
## for 1 year. These files are always timestamped in new releases,
## so it is safe to cache them for long periods of time.
<IfModule mod_expires.c>
	ExpiresActive On
	# Set default to 0 so .php/API requests are not cached
	ExpiresDefault A0

	# Do not cache MP4s, as Chrome and others tend to fail on first playback
	<FilesMatch "\.mp4$">
		<IfModule mod_headers.c>
			ExpiresActive Off
			Header set Cache-Control "private, no-cache, no-store, proxy-revalidate, no-transform"
			Header set Pragma "no-cache"
		</IfModule>
	</FilesMatch>

	# Console javascript and CSS is timestamped and can be aggressively cached
	<FilesMatch "console_.*\.(js|css)$">
		ExpiresByType text/javascript "access plus 1 year"
		ExpiresByType application/javascript "access plus 1 year"
		ExpiresByType application/x-javascript "access plus 1 year"
		ExpiresByType text/css "access plus 1 year"
	</FilesMatch>

	# Cached images are timestamped and can be aggressively cached
	<FilesMatch "\.\d{9,10}\.(jpg|jpeg|gif|png|JPG|JPEG|GIF|PNG)$">
		ExpiresDefault "access plus 1 year"
	</FilesMatch>
</IfModule>

## End Rewrite Rules

############################################
#######	 END KOKEN .htaccess rules  ########
############################################