Fetch content from external sources

[janekkkk]

Is your feature request related to a problem? Please describe.

I've got some articles in a database which can be accessed by a REST API. It would be great to access these articles from Nuxt Content.

Describe the solution you'd like

For example: const { title } = await this.$content('https://some-api/article-1').only(['title']).fetch()

Describe alternatives you've considered

Instead of accessing them like above. Download markdown files to the content directory and access them in the regular way.

Thanks for all the hard work and the awesome new features!

[benjamincanac]

Hey @janekkkk,

Nuxt Content parses markdown files on the file system on load and builds an API around the database generated from the content directory, it cannot be used on eternal sources.

You're alternative seems the way to go.

[cschweda]

An easy solution to this is to simply fetch the external markdown files in a node script that runs before the build/export steps.

Fetch (or use axios or whatever) the files, save them in the content directory, and then npm run build/export everything from there.

I do this -- and it seems to work (in my case, at least). I'm building on Netlify. No issues with this approach.

[janzheng]

An easy solution to this is to simply fetch the external markdown files in a node script that runs before the build/export steps.

Fetch (or use axios or whatever) the files, save them in the content directory, and then npm run build/export everything from there.

I do this -- and it seems to work (in my case, at least). I'm building on Netlify. No issues with this approach.

That's really interesting — if your external database updates (e.g. you have a new blog), how does that trigger Netlify to re-run the fetch script to build that new file into content directory, and then rebuild/re-export the Nuxt site?

[ninest]

I know the issue is closed, but I also wish there was a better way to use Nuxt Content's markdown features for external files. One solution is to use hmsk/frontmatter-markdown-loader, but I wish there was a way to do it using only Nuxt Content

[janzheng]

I know the issue is closed, but I also wish there was a better way to use Nuxt Content's markdown features for external files. One solution is to use hmsk/frontmatter-markdown-loader, but I wish there was a way to do it using only Nuxt Content

Here's a quick example of how to load and build the /content folder for Nuxt. This script runs independently of Nuxt, but can be added to the dev or build process as a script in package.json.

1. Create a loadContent.js script in your folder

const fetch = require("node-fetch");
const fs = require('fs');

console.log('>>> Loading External Content')

let url = 'https://your-url/api/content'
let path = './content/testfile.js'

const downloadFile = (async (url, path) => {
  const res = await fetch(url);
  const fileStream = fs.createWriteStream(path);
  await new Promise((resolve, reject) => {
    res.body.pipe(fileStream);
    res.body.on("error", (err) => {
      reject(err);
    });
    fileStream.on("finish", function() {
      resolve();
    });
  });
});

downloadFile(url,path)

2. In package.json create a script:

 "scripts": {
    ...
    "loadContent": "node helpers/loadContent.js",
    ...

3. In console, run npm run loadContent to load the files
4. To add to the build script, change the build script to

   "build": "npm run loadContent; nuxt build",

I don't add the content loader to the dev because my loader takes a while

Hope that helps!

[cschweda]

An easy solution to this is to simply fetch the external markdown files in a node script that runs before the build/export steps.
Fetch (or use axios or whatever) the files, save them in the content directory, and then npm run build/export everything from there.
I do this -- and it seems to work (in my case, at least). I'm building on Netlify. No issues with this approach.

That's really interesting — if your external database updates (e.g. you have a new blog), how does that trigger Netlify to re-run the fetch script to build that new file into content directory, and then rebuild/re-export the Nuxt site?

I use Strapi as an external API/auth service -- but the way I've dealt with this sort of issue with Strapi is to send a build hook when the user adds new content and then updates a "build" task.

My use case for this is when new content is added or updated -- I ask users to insert a new build task (a content-type in Strapi). When they enter the new task -- a note and their initials -- Strapi sends a webhook to Netlify to initiate a build. I have a node file that runs as part of the build to update the search index. This node script generates an index for the entire site (including the new content item(s)) -- and then rebuilds everything. (I use fuse.js for the search.)

This approach works great -- no issues. The only thing I need to watch are my Netlify build minutes (about 60 seconds for each site) -- but I'm managing about 15 sites this way on Netlify without issues. (These are low-traffic but critical sites -- so that's why Netlify is perfect.)

[txhai]

I am still looking for this. It would be great if I can host and update my blog posts on a Github repository, instead of rebuilding the entire Nuxt project.

[sillvva]

I also would like to be able to host my markdown files in Firebase Cloud Storage and fetch them at runtime via an API. I already figured out hot to dynamically generate the routes. The only thing I haven't figured out is how to parse the markdown into the tag. Some of my markdown files have vue components in them, so a 3rd party API isn't a feasible option for me.

[goodpixels]

Are you using a webook or a server hook? Would you be able to share some code if you don't mind please?