Feature/seo

[corrie-sloot]

Fulfilment of mPowered’s DAO, Round 13 relating to search engine optimisations (SEO).

Changes:

• remove bug which adds noindex,nofollow meta tag to ALL pages, now search engines will crawl the market
• add a third party gatsby plugin which generates a robot.txt file on build with a sitemap
• add a meta description to every asset page, which comes from the first 150 characters of the description markdown
• only show the ocean twitter meta tag when in the ocean marketplace

Note:
This PR is dependant on a change to aquarius. The robots.txt file refers to a new endpoint which outputs an XML sitemap of all appropriate asset URLs. Please do not merge until the feature/seo branch is merged into aquarius and implemented in production.

[vercel]

@corrie-sloot is attempting to deploy a commit to the Ocean Protocol Team on Vercel.

A member of the Team first needs to authorize it.

[codeclimate]

Code Climate has analyzed commit 6b02b6f and detected 0 issues on this pull request.

View more on Code Climate.

[kremalicious]

sorry, we are not going to accept more features on the v3 code base which currently is main. Would suggest adding this against v4 branch which does not use Gatsby anymore and where this change might not be needed.

Also not sure if a change to aquarius is needed for that, as you can generate the robots.txt or sitemap.xml during build process of market alone. All this then also depends on constant rebuilds of market, as assets published after last deployment would be not included. Likewise, I am not sure the intended title& description for every asset details page would work, as that's only available after some fetching on client-side for all asset details pages.

To me, if we want to really make sure all published assets are properly indexed we need to pre-render all asset details pages based on querying Aquarius on build time, then on client-side re-hydrate what we show by querying Aquarius again. Both Gatsby & Next.js allow this to be done but it can become complex and locks us into one framework so we always kept the app as a lightweight client-side fetching app.

But making sure all existing pre-rendered pages index properly, especially front page (cool bug!) is already super valuable so would reduce the scope against v4

[corrie-sloot]

@kremalicious , might I suggest updating market.oceanprotocol.com at least with the fix for the noindex,nofollow.

It will have a significant impact on making the market searchable and the sooner it is done, the more traffic can get to the market.