sitemap not being included in robots.txt

[myusuf3]

I have tried both default discovery as well as specifying it manually using the ROBOTS_SITEMAP_URLS setting but I am still unable to get the site map to show up on the robots.txt

[andreyshipilov]

Same here. Doesn't resolve or appear in "robots.txt".

[andreyshipilov]

In my case it looks like rverse() doesn't respect the cache_page() decorator.

# This one doesn't work.
url(r"^sitemap\.xml$", cache_page(60 * 60)("django.contrib.sitemaps.views.sitemap"), {"sitemaps": SITEMAPS}),

This one does.
url(r"^sitemap\.xml$", "django.contrib.sitemaps.views.sitemap", {"sitemaps": SITEMAPS}),

[yakky]

@myusuf3 @Tezro I cannot reproduce this, could you provide more details?

[myusuf3]

woo! wayback machine here. I dont even remember creating this issue.

[SalahAdDin]

👍

[mheppner]

I know this is a pretty old issue, but I wanted to clarify that reverse() does not pick up on the sitemap if it's behind cache_page() in your URL conf. It might be helpful for others to update the documentation. In my case, I was using a main index (django.contrib.sitemaps.views.index) which linked off to the sections (django.contrib.sitemaps.sitemap), so I just took off the caching for the index and django-robots could then reverse to it. The sections can still be cached.

Alternatively, if you just had 1 sitemap, it seems to work fine if you create an alias view and don't use cache_page() directly in your URL conf:

# views.py
from django.contrib.sitemaps import views as sitemaps_views
from django.views.decorators.cache import cache_page

@cache_page(10 * 60)
def sitemap_index(*args, **kwargs):
    return sitemaps_views.index(*args, **kwargs)

[yakky]

@mheppner when merged, could you check if the fix (which is under way) for #61 solves the cache issue?

[myusuf3]

4 years guys.

[jezdez]

@myusuf3 For the record, that type of snark is counter-productive and I think you should know better. Please be so kind and stop giving this type of feedback.

[mheppner]

I sure can @yakky. ETA on the merge? The one you linked is just the issue, did you mean #71?

[yakky]

@mheppner actually this commit ba2d5b6 sorry
When I wrote the above comment I thought I would have opened a Pr specific for the deprecation but it ended up with the whole general Django version update
I would like to wrap everything by the end of the week

[mheppner]

@yakky the link to that commit doesn't work either lol. I'll check out your branch this evening.

[yakky]

@mheppner 🤦‍♂️

[mheppner]

@yakky I tried the features/djangos branch and that didn't seem to change anything. Everything else was fine though. I think the problem me and @andreyshipilov were having is just how cache_page() behaves. That answer is from 2012, so maybe something has changed, but I can't find anything.

What about adding a setting to django-robots for specifying a named URL to reverse to?

[yakky]

@mheppner I'm not super fond of adding settings, but it seems adequate to cover this case

[yakky]

@mheppner @andreyshipilov could you please check the linked pr?

[mheppner]

@yakky looks good to me, thanks! 😃

[yakky]

@mheppner thanks for the review!

[andreyshipilov]

@yakky Works for me too :)

[yakky]

@andreyshipilov 👍 thanks

[SalahAdDin]

@yakky For when have you planned release a new version?

[yakky]

@SalahAdDin with a little luck this weekend

[SalahAdDin]

@yakky, oh i hope!

[yakky]

@SalahAdDin 3.0rc1 has been released on Pypi. I'm going to check if everything is ok and I will release 3.0 final in a couple of days (sooner if you could confirm rc1 is fine! ;) )

[SalahAdDin]

Man, i have a problem using the new url version:

This argument is the problem: {'sitemaps': []}.
I think is a problem because i'm using wagtail's sitemap view. So, deleting this argument cached view works very well.

[mheppner]

@SalahAdDin Can you give more information about this? Where is that argument you're talking about? For what it's worth, Wagtail doesn't use Django's sitemap framework, they're rolling their own, which means django-robots won't be reversing to it automatically. Can you try giving your URL a name and setting it to ROBOTS_SITEMAP_VIEW_NAME?

[SalahAdDin]

@mheppner Yes, i did it, i only delete these argument, and works fine.

[yakky]

@SalahAdDin @mheppner thanks for helping with this. Would you check if linked PR clarifies how to correctly configure robots in the case above?

[mheppner]

Ah, now I see what @SalahAdDin was talking about. @yakky, yeah I think that clears it up 👍

[mheppner]

@yakky I just tried 3.0rc1 and everything seems to be working, thanks! 🥇