204 - No content #14

Buom01 · 2015-08-03T15:27:37Z

Hi,

I have a probleme with my website, what I try to do, my pages return nothing with a 204 HTTP status code with ?_escaped_fragment_=, else, the 404 page send correctly the 404 HTTP status code, but is also empty

Notes:

My host is openshift, I have a free plan
My host has an Apache proxy for routing <-- Can be the probleme ?
I use demeteorizer to convert to simple nodejs app
I used npm to install phantomjs

Code:

spiderable-test.js

if(Meteor.isServer){
  Spiderable.debug = true;
  Spiderable.customQuery = true;
}
Router.configure({
  notFoundTemplate: '_404'
});
Router.plugin('dataNotFound', {notFoundTemplate: '_404'});

Router.route('/', function () {
  this.render('main');
});

Router.onAfterAction(function(){
  if(this.ready()){ 
    Meteor.isReadyForSpiderable = true;
  }
});

console.log('Scripting...');

spiderable-test.html

<head>
  <title>spiderable-test</title>
</head>

<body>
</body>

<template name="main">
  <h1>Page d'accueil</h1>
</template>
<template name="_404">
  <h1>404</h1>
  <h3>Oops, page not found</h3>
  <p>Sorry, page you're requested is not exists or was deleted</p>
</template>

Mongo content:

Complete informations

[test-buom01.rhcloud.com ******]\> mongo $OPENSHIFT_MONGODB_URLtest
MongoDB shell version: 2.4.9                                                                                                                                                                               
connecting to: 127.*.*.*:27017/admin                                                                                                                                                                  
Welcome to the MongoDB shell.                                                                                                                                                                              
For interactive help, type "help".                                                                                                                                                                         
For more comprehensive documentation, see                                                                                                                                                                  
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
> use test
switched to db test
> db.SpiderableCacheCollection.find({})
{ "_id" : "yLhw4TicXDzEeT8E5", "hash" : "f807c97ecbb6087b674f75cc45e714675b8cdd0b56959f7600c89ebc860f9678", "url" : "http://test-buom01.rhcloud.com/?___isRunningPhantomJS___=true", "headers" : [      { "name" : "Date",         "value" : "Mon, 03 Aug 2015 13:41:14 GMT" },    {       "name" : "Vary",        "value" : "Accept-Encoding" },  {       "name" : "Content-Type",        "value" : "text/html; charset=utf-8" },    {       "name" : "Content-Encoding",    "value" : "gzip" },     {       "name" : "Keep-Alive",  "value" : "timeout=15, max=100" },      {       "name" : "Connection",  "value" : "Keep-Alive" } ], "content" : "<!DOCTYPE html><html><head>\n  <link rel=\"stylesheet\" type=\"text/css\" class=\"__meteor-css__\" href=\"/20ae2c8d51b2507244e598844414ecdec2615ce3.css\">\n\n\n\n\n  \n\n\n\n\n<title>spiderable-test</title>\n</head>\n<body>\n\n\n\n<h1>Page d'accueil</h1></body></html>", "status" : 204, "createdAt" : ISODate("2015-08-03T13:41:15.360Z") }
>

Just its content

{
   "_id":"yLhw4TicXDzEeT8E5",
   "hash":"f807c97ecbb6087b674f75cc45e714675b8cdd0b56959f7600c89ebc860f9678",
   "url":"http://test-buom01.rhcloud.com/?___isRunningPhantomJS___=true",
   "headers":[
      {
         "name":"Date",
         "value":"Mon, 03 Aug 2015 13:41:14 GMT"
      },
      {
         "name":"Vary",
         "value":"Accept-Encoding"
      },
      {
         "name":"Content-Type",
         "value":"text/html; charset=utf-8"
      },
      {
         "name":"Content-Encoding",
         "value":"gzip"
      },
      {
         "name":"Keep-Alive",
         "value":"timeout=15, max=100"
      },
      {
         "name":"Connection",
         "value":"Keep-Alive"
      }
   ],
   "content":"<!DOCTYPE html><html><head>\n  <link rel=\"stylesheet\" type=\"text/css\" class=\"__meteor-css__\" href=\"/20ae2c8d51b2507244e598844414ecdec2615ce3.css\">\n\n\n\n\n  \n\n\n\n\n<title>spiderable-test</title>\n</head>\n<body>\n\n\n\n<h1>Page d'accueil</h1></body></html>",
   "status":204,
   "createdAt":   ISODate("2015-08-03T13:41:15.360   Z")
}

My building home made script:

.build2.sh

./.compress2.sh
export MINI="./.mini/nodejs"
echo "{{{{BUILDING}}}}"
cd $MINI
echo "Remove output"
chmod -R 777 ./.demeteorized
rm -R ./.demetorized
echo "Build and demeteorization..."
demeteorizer
echo "Fix permissions again..."
chmod -R 777 ./.demeteorized

cd ./.demeteorized

echo "Adding env vars..."

# settings.json is actually empty
echo "process.env.METEOR_SETTINGS = '$(echo $(cat ../settings.json))';
$(cat ./main.js)" > ./main.js 


#sed -i '1i process.env.MAIL_URL = "smtp://*********/";' ./main.js
#sed -i '1i process.env.ROOT_URL = ("http://" + process.env.OPENSHIFT_APP_DNS) || "http://localhost:8000"' ./main.js
#sed -i '1i process.env.MONGO_URL = (process.env.OPENSHIFT_MONGODB_DB_URL + process.env.OPENSHIFT_APP_NAME) || "mongodb://localhost:27017/meteor";' ./main.js
#sed -i '1i process.env.PORT = process.env.OPENSHIFT_NODEJS_PORT || 8000;' ./main.js
#sed -i '1i process.env.BIND_IP = process.env.OPENSHIFT_NODEJS_IP || "127.0.0.1";' ./main.js



#sed -i '1i process.env.MAIL_URL = "smtp://*******/";' ./main.js
sed -i '1i process.env.ROOT_URL = "http://" + (process.env.OPENSHIFT_APP_DNS || "localhost:8000");' ./main.js
#sed -i '1i process.env.ROOT_URL = "http://"+ process.env.OPENSHIFT_NODEJS_IP + ":" + process.env.OPENSHIFT_NODEJS_PORT;' ./main.js
sed -i '1i process.env.MONGO_URL = (process.env.OPENSHIFT_MONGODB_DB_URL + process.env.OPENSHIFT_APP_NAME) || "mongodb://localhost:27017/meteor";' ./main.js
sed -i '1i process.env.PORT = process.env.OPENSHIFT_NODEJS_PORT || 8000;' ./main.js
sed -i '1i process.env.BIND_IP = process.env.OPENSHIFT_NODEJS_IP || "127.0.0.1";' ./main.js


echo "Copying into git directory..."
cd ../../..
rm -R ./.end/*
#mkdir ./.end
#cp -R ./.static-files/.git ./.end/
#cp -R ./.static-files/.openshift ./.end/
cp -R $MINI/.demeteorized/* ./.end/
echo "Done"

.compress2.sh

echo "{{{{COMPRESSION}}}}"
export MINI="./.mini/nodejs"
echo "Fix permissions..."
chmod -R 777 $MINI
echo "Removing..."
rm -R $MINI
echo "Copying..."
mkdir $MINI
cp -R * $MINI
cp -R ./.meteor $MINI
echo "Minifying..."
#htmlminify -o $MINI/spiderable-test.html $MINI/spiderable-test.html
echo "Remove enters..."
#echo  $(cat $MINI/spiderable-test.html)>$MINI/spiderable-test.html
echo "Done"

.deploy2.sh

cd .end
git add .
git commit -m "Update"
git push

Packages

gadicohen:phantomjs 0.0.2 phantomjs installation script; wrapper for node phantomjs package + Sun
iron:router 1.0.9 Routing specifically designed for Meteor
jazeee:spiderable-longer-timeout 1.2.5 Extended spiderable package: SSL, caching, longer timeout, no stdin issues, publish flag
meteor-platform 1.2.2 Include a standard set of Meteor packages in your app

Se it yourselft

http://test-buom01.rhcloud.com/
http://test-buom01.rhcloud.com/?_escaped_fragment_=

Conclusion

Anybody can help me ?
What can I do ?

Thank you for reading, and sorry if I have a bad English

The text was updated successfully, but these errors were encountered:

jazeee · 2015-08-03T16:40:25Z

For now, revert to jazeee:spiderable-longer-timeout 1.2.2 which works, but does not handle 404s.
We will have to look at it.
@dr-dimitru have you seen this issue?

dr-dimitru · 2015-08-03T16:50:16Z

@Buom01 At first try to access node app directly without proxy, could you provide a link (should be with port, like 3000, actually I've tried 300 port but got no response)?
BTW (see image below) it is something wrong with your setup, your server not supposed to response with 204

dr-dimitru · 2015-08-03T16:50:40Z

@jazeee I've never meet such issue before

Buom01 · 2015-08-03T19:39:49Z

Sadly, when I talked about proxy, I wanted to say a reverse proxy created by my hoster to get the 80 port (shared hosting...).

BTW (see image below) it is something wrong with your setup, your server not supposed to response with 204

... Now I remeber that my hoster was recently under maintenance, and I think that they had changed their proxy config, (...).

Solution found !
I changed ROOT_URL to nodejs directly like it;

process.env.ROOT_URL = "http://" + process.env.OPENSHIFT_NODEJS_IP +":"+process.env.OPENSHIFT_NODEJS_PORT';

And not like (not to http://test-buom01.rhcloud.com/)

process.env.ROOT_URL = "http://" + (process.env.OPENSHIFT_APP_DNS || "localhost:8000");

An idea is to get it from $PORT and $BIND_IP env variables and not from $ROOT_URL. (I dont know the default value of $ROOT_URL)

~~I keep this app in this status to notify my hoster that there are bugs with they apache proxy's config~~

Thank you very much 👍

jazeee · 2015-08-03T20:13:00Z

Ok, sounds like it was a configuration/maintenance issue.
Thanks.

jazeee · 2015-08-10T15:40:53Z

I found that this issue does exist, and is due to a bug in the phantomjs script.

It appears to depend on whether one uses a proxy, or similar. For example, if one uses nginx in front of meteor, for SSL services, for example.

One way to test is, create a meteor server, with nginx wrapper for SSL. Then, test the phantomjs script:

phantomjs --load-images=no --ssl-protocol=TLSv1 --ignore-ssl-errors=true --web-security=false jazeee-meteor-spiderable/lib/phantom_script.js https://your-server

This returns a 204. The reason is that https://github.com/jazeee/jazeee-meteor-spiderable/blob/master/lib/phantom_script.js#L60 is not correct.

page.onResourceReceived(...) triggers on any resource request, including image assets or Meteor Websocket connection. In the first part of that function, we correctly check for the URL, however, for the remaining part, we accept any status. Since this seems unnecessary, I am removing that part to fix this bug.

EDIT:
When I test the current script, I see the URL: https://.../sockjs/383/l2de9dnl/xhr_send, with status 204, which is the wrong status for the page. If I remove the problem code, I get the right result.

dr-dimitru · 2015-08-10T15:47:22Z

Without this logic statement you will not receive status code on redirects

On 10 Aug 2015, at 18:40, Jaz notifications@github.com wrote:

I found that this issue does exist, and is due to a bug in the phantomjs script.

It appears to depend on whether one uses a proxy, or similar. For example, if one uses nginx in front of meteor, for SSL services, for example.

One way to test is, create a meteor server, with nginx wrapper for SSL. Then, test the phantomjs script:

phantomjs --load-images=no --ssl-protocol=TLSv1 --ignore-ssl-errors=true --web-security=false jazeee-meteor-spiderable/lib/phantom_script.js https://your-server
This returns a 204. The reason is that https://github.com/jazeee/jazeee-meteor-spiderable/blob/master/lib/phantom_script.js#L60 is not correct.

page.onResourceReceived(...) triggers on any resource request, including image assets or Meteor Websocket connection. In the first part of that function, we correctly check for the URL, however, for the remaining part, we accept any status. Since this seems unnecessary, I am removing that part to fix this bug.

—
Reply to this email directly or view it on GitHub.

dr-dimitru · 2015-08-10T15:50:07Z

I believe developer should workaround with proxy server, cause we are using nginx and SSL on our production stage with this package without any issues.

On 10 Aug 2015, at 18:47, Dmitriy A. Golev dr.dimitru@gmail.com wrote:

Without this logic statement you will not receive status code on redirects

On 10 Aug 2015, at 18:40, Jaz notifications@github.com wrote:

I found that this issue does exist, and is due to a bug in the phantomjs script.

It appears to depend on whether one uses a proxy, or similar. For example, if one uses nginx in front of meteor, for SSL services, for example.

One way to test is, create a meteor server, with nginx wrapper for SSL. Then, test the phantomjs script:

phantomjs --load-images=no --ssl-protocol=TLSv1 --ignore-ssl-errors=true --web-security=false jazeee-meteor-spiderable/lib/phantom_script.js https://your-server
This returns a 204. The reason is that https://github.com/jazeee/jazeee-meteor-spiderable/blob/master/lib/phantom_script.js#L60 is not correct.

page.onResourceReceived(...) triggers on any resource request, including image assets or Meteor Websocket connection. In the first part of that function, we correctly check for the URL, however, for the remaining part, we accept any status. Since this seems unnecessary, I am removing that part to fix this bug.

—
Reply to this email directly or view it on GitHub.

jazeee · 2015-08-10T16:06:55Z

I don't think it makes sense to allow rely on any arbitrary resource response code. For example, as part of the processing, I see a response code of 204, for URL: https://.../sockjs/383/l2de9dnl/xhr_send

This is the last URL that is processed before phantomJS finishes. It is quite complex to debug these issues, and other users will have serious problems that will appear random. For example, it works locally, but doesn't work due to some arbitrary nginx configuration, version, or other arbitrary server setup/load etc.

Since this issue affects the majority of normal cases, we will have to look at other options for redirects.
For example, if it is specifically handling redirects, then we should capture that in a different way, and handle it specifically.

dr-dimitru · 2015-08-10T16:15:24Z

I believe it should work with any kind of redirects. BTW http-redirect is handled by response.redirectURL. But partly I agree with you, so we should pre-program support for "complete" responses, like 200, 302, 404, 400, 500, etc., and avoid "non-complete", like 206, 204, etc.
And add all supported codes into docs, what do you think?

jazeee · 2015-08-10T18:28:53Z

Quite possible. My changes seem to also break 404, but I think we will have to be very careful about these issues. (In a separate ticket).
I wouldn't want phantom to respond with a 404 just because an image file or asset is missing.

Of course, the primary goal is to make the successful paths work, in most people's scenarios.

jazeee · 2015-08-10T19:02:57Z

Just a note, if I log all response statuses during a redirect test, within onResourceReceived, I get about 15x200, then 204 then 4x200 then 204, then 200. These are most likely due to the various JavaScript pages or websocket connections occurring during the page load. I don't see any redirect status codes. If I test the same, but directly to meteor, bypassing nginx, I see about 12x200 codes.

Not sure if it is a PhantomJS issue, a polling issue, or something else. In any case, I doubt that we can count on the intermediate status codes being representative of the page's final code.

In reality, I think we have to specify something else, such as adding Spiderable.responseCode = 404, to the IronRouter/Meteor code.

dr-dimitru · 2015-08-10T22:15:03Z

It is easily solved at this statement
The line you have removed (maybe was incorrect) shouldn't be removed, it should be replaced with something more complex to handle redirects (when page's URL is not equal to original requested page), now you broke correct redirects and all kind of responses (except 200 (which is set by default)).

jazeee closed this as completed Aug 3, 2015

jazeee reopened this Aug 10, 2015

jazeee added a commit that referenced this issue Aug 10, 2015

Fixing issue with nginx and spurious errors. #14

afc12ad

jazeee added a commit that referenced this issue Aug 10, 2015

Fixing issue with nginx and spurious errors. #14

322cb5a

jazeee added a commit that referenced this issue Aug 10, 2015

Fixing issue with nginx and spurious errors. #14

9f907a9

jazeee closed this as completed Aug 10, 2015

jazeee mentioned this issue Aug 10, 2015

I should get reliably get a 404 when spidering an invalid page #15

Closed

jazeee added a commit that referenced this issue Aug 10, 2015

Fixing issue with nginx and spurious errors. #14

603292f

Buom01 mentioned this issue Nov 5, 2015

Uploaded image 404'ing after a few days tomitrescak/meteor-uploads#113

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

204 - No content #14

204 - No content #14

Buom01 commented Aug 3, 2015

jazeee commented Aug 3, 2015

dr-dimitru commented Aug 3, 2015

dr-dimitru commented Aug 3, 2015

Buom01 commented Aug 3, 2015

jazeee commented Aug 3, 2015

jazeee commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015

jazeee commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015

jazeee commented Aug 10, 2015

jazeee commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015

204 - No content #14

204 - No content #14

Comments

Buom01 commented Aug 3, 2015

Hi,

Notes:

Code:

spiderable-test.js

spiderable-test.html

Mongo content:

Complete informations

Just its content

My building home made script:

.build2.sh

.compress2.sh

.deploy2.sh

Packages

Se it yourselft

Conclusion

jazeee commented Aug 3, 2015

dr-dimitru commented Aug 3, 2015

dr-dimitru commented Aug 3, 2015

Buom01 commented Aug 3, 2015

jazeee commented Aug 3, 2015

jazeee commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015

jazeee commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015

jazeee commented Aug 10, 2015

jazeee commented Aug 10, 2015

dr-dimitru commented Aug 10, 2015