robots.txt block crawl from my components

[triplecasquette]

As said in the title the default robots.txt block the google crawler getting info in my Navbar and footer component in NextJS. On pages the build solve the problem. But how far I know it's impossible to getStaticProps on components in NextJs and so build informations. That's why I need to have still no indexation of my api, but authorize crawling. where's the robot or where to config?
I use a self hosted, so i found (I think) the public folder in /@directus/app/dist but do I have to put a robot here or ... ? I don't know
Thank for any reply

[br41nslug]

Extending the robots.txt is an open issue im afraid #15421 however there is a workaround extension in the issue comments.

[br41nslug]

Indexing is disabled by default you'll want to take that robots.txt and allow the bots you want to be able to index :)

[triplecasquette]

I read somewhere robot is only for crawling not for index. Disable if crawling is disable obviously. But I want google to be permitted to crawl and be able to get data from my front, but not to be able to index directus obviously too. Is there a way? Because my robot is now user-agent * Allow /

[br41nslug]

I am not super familiar with the specific syntax of robots.txt as i tend to just block everthing 😬 Imo crawling and indexing are 2 sides of the same coin where crawling is the act of moving through all reachable pages and indexing is saving the resulting data. What specifically do you want to allow and what do you want to block?

[triplecasquette]

Actually I would have preferred to let the default disallow. But as I said I see on google search console that my networks, and contact info stored in directus and reach via react on my nextJS app can't be accessed by google. Every other thing are build with getStaticProps method of next but as my navbar and footer are component I cannot use this method. So I would want these informations to be visible for google but my directus not indexed. I hope I'm clear.

[triplecasquette]

Forget about it, I've done it another way. Rule number one of the code: If it doesn't work, do it another way 😅 I get back my robot to disallow all and succeed in finding a way to build my infos and networks globally. Thank you for your time. I love you all directus Team. Great project, great member, available... Never change you are all awesome I LOVE DIRECTUS !!!! 😄