Skip to content
Permalink
Browse files

First

  • Loading branch information...
dariusk committed Oct 15, 2018
0 parents commit 36bb506bcb9cd8b04d6a5bb2f74d871c6ff9ba21
Showing with 799 additions and 0 deletions.
  1. +4 −0 .gitignore
  2. +22 −0 LICENSE-MIT
  3. +80 −0 README.md
  4. +7 −0 config.json.template
  5. +55 −0 index.js
  6. +26 −0 package.json
  7. +88 −0 public/convert/index.html
  8. +105 −0 routes/api.js
  9. +95 −0 routes/inbox.js
  10. +8 −0 routes/index.js
  11. +40 −0 routes/user.js
  12. +23 −0 routes/webfinger.js
  13. +197 −0 updateFeeds.js
  14. +11 −0 views/home.pug
  15. +23 −0 views/style.css
  16. +15 −0 views/user.pug
@@ -0,0 +1,4 @@
node_modules/
*.db
package-lock.json
config.json
@@ -0,0 +1,22 @@
Copyright (c) 2018 Darius Kazemi

Permission is hereby granted, free of charge, to any person
obtaining a copy of this software and associated documentation
files (the "Software"), to deal in the Software without
restriction, including without limitation the rights to use,
copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following
conditions:

The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
OTHER DEALINGS IN THE SOFTWARE.
@@ -0,0 +1,80 @@
# RSS to ActivityPub Converter

This is a server that lets users convert any RSS feed to an ActivityPub actor that can be followed by users on ActivityPub-compliant social networks like Mastodon. For a demo of this in action, see https://bots.tinysubversions.com/convert/

## Requirements

This requires Node.js v10.10.0 or above.

## Installation

Clone the repository, then `cd` into its root directory. Install dependencies:

`npm i`

Then copy `config.json.template` to `config.json`:

`cp config.json.template config.json`

Update your new `config.json` file:

```js
{
"DOMAIN": "mydomain.com",
"PORT_HTTP": "3000",
"PORT_HTTPS": "8443",
"PRIVKEY_PATH": "/path/to/your/ssl/privkey.pem",
"CERT_PATH": "/path/to/your/ssl/cert.pem"
}
```

`DOMAIN`: your domain! this should be a discoverable domain of some kind like "example.com" or "rss.example.com"
`PORT_HTTP`: the http port that Express runs on
`PORT_HTTPS`: the https port that Express runs on
`PRIVKEY_PATH`: point this to your private key you got from Certbot or similar
`CERT_PATH`: point this to your cert you got from Certbot or similar

Run the server!

`node index.js`

Go to `https://whateveryourdomainis.com:3000/convert` or whatever port you selected for HTTP, and enter an RSS feed and a username.If all goes well it will create a new ActivityPub user with instructions on how to view the user.

## Sending out updates to followers

There is also a file called `updateFeeds.js` that needs to be run on a cron job or similar scheduler. I like to run mine once a minute. It queries every RSS feed in the database to see if there has been a change to the feed. If there is a new post, it sends out the new post to everyone subscribed to its corresponding ActivityPub Actor.

## Local testing

You can use a service like [ngrok](https://ngrok.com/) to test things out before you deploy on a real server. All you need to do is install ngrok and run `ngrok http 3000` (or whatever port you're using if you changed it). Then go to your `config.json` and update the `DOMAIN` field to whatever `abcdef.ngrok.io` domain that ngrok gives you and restart your server.

Then make sure to manually run `updateFeed.js` when the feed changes. I recommend having your own test RSS feed that you can update whenever you want.

## Database

This server uses a SQLite database to keep track of all the data. There are two tables in the database: `accounts` and `feeds`.

### `accounts`

This table keeps track of all the data needed for the accounts. Columns:

* `name` `TEXT PRIMARY KEY`: the account name, in the form `thename@example.com`
* `privkey` `TEXT`: the RSA private key for the account
* `pubkey` `TEXT`: the RSA public key for the account
* `webfinger` `TEXT`: the entire contents of the webfinger JSON served for this account
* `actor` `TEXT`: the entire contents of the actor JSON served for this account
* `apikey` `TEXT`: the API key associated with this account
* `followers` `TEXT`: a JSON-formatted array of the URL for the Actor JSON of all followers, in the form `["https://remote.server/users/somePerson", "https://another.remote.server/ourUsers/anotherPerson"]`
* `messages` `TEXT`: not yet used but will eventually store all messages so we can render them on a "profile" page

### `feeds`

This table keeps track of all the data needed for the feeds. Columns:

* `feed` `TEXT PRIMARY KEY`: the URI of the RSS feed
* `username` `TEXT`: the username associated with the RSS feed
* `content` `TEXT`: the most recent copy fetched of the RSS feed's contents

## License

Copyright (c) 2018 Darius Kazemi. Licensed under the MIT license.
@@ -0,0 +1,7 @@
{
"DOMAIN": "",
"PORT_HTTP": "3000",
"PORT_HTTPS": "8443",
"PRIVKEY_PATH": "",
"CERT_PATH": ""
}
@@ -0,0 +1,55 @@
const config = require('./config.json');
const { DOMAIN, PRIVKEY_PATH, CERT_PATH, PORT_HTTP, PORT_HTTPS } = config;
const express = require('express');
const app = express();
const Database = require('better-sqlite3');
const db = new Database('bot-node.db');
const fs = require('fs');
const routes = require('./routes'),
bodyParser = require('body-parser'),
cors = require('cors'),
http = require('http');
let sslOptions;

try {
sslOptions = {
key: fs.readFileSync(PRIVKEY_PATH),
cert: fs.readFileSync(CERT_PATH)
};
} catch(err) {
if (err.errno === -2) {
console.log('No SSL key and/or cert found, not enabling https server');
}
else {
console.log(err);
}
}

// if there is no `accounts` table in the DB, create an empty table
db.prepare('CREATE TABLE IF NOT EXISTS accounts (name TEXT PRIMARY KEY, privkey TEXT, pubkey TEXT, webfinger TEXT, actor TEXT, apikey TEXT, followers TEXT, messages TEXT)').run();
// if there is no `feeds` table in the DB, create an empty table
db.prepare('CREATE TABLE IF NOT EXISTS feeds (feed TEXT PRIMARY KEY, username TEXT, content TEXT)').run();

app.set('db', db);
app.set('domain', DOMAIN);
app.set('port', process.env.PORT || PORT_HTTP);
app.set('port-https', process.env.PORT_HTTPS || PORT_HTTPS);
app.set('views', './views');
app.set('view engine', 'pug');
app.use(bodyParser.json({type: 'application/activity+json'})); // support json encoded bodies
app.use(bodyParser.urlencoded({ extended: true })); // support encoded bodies

app.get('/', (req, res) => res.render('home'));

// admin page
app.options('/api', cors());
app.use('/api', cors(), routes.api);
app.use('/admin', express.static('public/admin'));
app.use('/convert', express.static('public/convert'));
app.use('/.well-known/webfinger', cors(), routes.webfinger);
app.use('/u', cors(), routes.user);
app.use('/api/inbox', cors(), routes.inbox);

http.createServer(app).listen(app.get('port'), function(){
console.log('Express server listening on port ' + app.get('port'));
});
@@ -0,0 +1,26 @@
{
"name": "bot-node",
"version": "1.0.0",
"description": "",
"main": "index.js",
"dependencies": {
"better-sqlite3": "^5.0.1",
"body-parser": "^1.18.3",
"cheerio": "^1.0.0-rc.2",
"cors": "^2.8.4",
"express": "^4.16.3",
"generate-rsa-keypair": "^0.1.2",
"pug": "^2.0.3",
"request": "^2.87.0",
"rss-parser": "^3.4.3"
},
"engines": {
"node": ">=10.10.0"
},
"devDependencies": {},
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1"
},
"author": "",
"license": "MIT"
}
@@ -0,0 +1,88 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Convert an RSS feed to ActivityPub</title>
<style>
body {
font-family: sans-serif;
max-width: 900px;
margin: 30px;
}
img {
max-width: 100px;
}
li {
margin-bottom: 0.2em;
}
.account {
}
input {
width: 300px;
font-size: 1.2em;
}
.hint {
font-size: 0.8em;
}
button {
font-size: 1.2em;
}
</style>
</head>
<body>
<h1>Convert an RSS feed to ActivityPub</h1>
<p>Put the full RSS feed URL in here, and pick a username for the account that will track the feed.</p>
<p>
<input id="feed" type="text" placeholder="https://example.com/feed.xml"/>
</p>
<p>
<input id="username" type="text" placeholder="username"/><br><span class="hint">only letters, digits, and underscore (_) allowed</span>
</p>
<button onclick="submit()">Submit</button>
<div id="out">
</div>

<script>
// https://bots.tinysubversions.com/api/convert/?feed=https://toomuchnotenough.site/feed.xml&username=tmne
function submit() {
let feed = document.querySelector('#feed').value;
let username = document.querySelector('#username').value;
let out = document.querySelector('#out');
fetch(`/api/convert/?feed=${feed}&username=${username}`)
.then(function(response) {
if (response.status !== 200) {
out.innerHTML = `<p>Error: ${JSON.stringify(response.statusText)}</p>`;
return {};
}
return response.json();
})
.then(function(myJson) {
console.log((myJson));
// a feed exists in the database
if (myJson.content) {
// was it a match on feed
if (myJson.feed === feed) {
console.log('feed match!');
out.innerHTML = `<p>This feed already exists! Follow @${myJson.username}@bots.tinysubversions.com.</p>`;
window.location = `/u/${myJson.username}`;
}
// was it a match on username
else if (myJson.username === username) {
console.log('username match!');
out.innerHTML = `<p>This username is already taken for <a href="${myJson.feed}">this feed</a>.</p>`;
}
}
else if (myJson.title) {
out.innerHTML = `<p>Okay! There is now an ActivityPub actor for ${myJson.title}. You should be able to search for it from your ActivityPub client (Mastodon, Pleroma, etc) using this identifier: @${username}@bots.tinysubversions.com. You won't see anything there until the next time the RSS feed updates. You can check out the profile page for this feed at <a href="https://bots.tinysubversions.com/u/${username}/">https://bots.tinysubversions.com/u/${username}</a> too!</p>`;
}
})
.catch(error => {
console.log('!!!',error);
out.innerHTML = `<p>Error: ${error}</p>`;
});
}
</script>
</body>
</html>
@@ -0,0 +1,105 @@
'use strict';
const express = require('express'),
router = express.Router(),
crypto = require('crypto'),
Parser = require('rss-parser'),
generateRSAKeypair = require('generate-rsa-keypair');

router.get('/convert', function (req, res) {
let db = req.app.get('db');
console.log(req.query);
let username = req.query.username;
let feed = req.query.feed;
// reject if username is invalid
if (username.match(/^[a-zA-Z0-9_]+$/) === null) {
return res.status(400).json('Invalid username! Only alphanumerics and underscore (_) allowed.');
}
// check to see if feed exists
let result = db.prepare('select * from feeds where feed = ? or username = ?').get(feed, username);
// see if we already have an entry for this feed
if (result) {
// return feed
res.status(200).json(result);
}
else if(feed && username) {
console.log('VALIDATING');
// validate the RSS
let parser = new Parser();
parser.parseURL(feed, function(err, feedData) {
if (err) {
res.status(400).json({err: err.message});
}
else {
console.log(feedData.title);
console.log('end!!!!');
res.status(200).json(feedData);
let displayName = feedData.title;
let account = username;
// create new user
let db = req.app.get('db');
let domain = req.app.get('domain');
// create keypair
var pair = generateRSAKeypair();
let imageUrl = null;
// if image exists set image
if (feedData.image && feedData.image.url) {
imageUrl = feedData.image.url;
}
let actorRecord = createActor(account, domain, pair.public, displayName, imageUrl);
let webfingerRecord = createWebfinger(account, domain);
const apikey = crypto.randomBytes(16).toString('hex');
db.prepare('insert or replace into accounts(name, actor, apikey, pubkey, privkey, webfinger) values(?, ?, ?, ?, ?, ?)').run( `${account}@${domain}`, apikey, pair.public, pair.private, JSON.stringify(actorRecord), JSON.stringify(webfingerRecord));
let content = JSON.stringify(feedData);
db.prepare('insert or replace into feeds(feed, username, content) values(?, ?, ?)').run( feed, username, content);
}
});
}
else {
res.status(404).json({msg: 'unknown error'});
}
});

function createActor(name, domain, pubkey, displayName, imageUrl) {
displayName = displayName || name;
let actor = {
'@context': [
'https://www.w3.org/ns/activitystreams',
'https://w3id.org/security/v1'
],

'id': `https://${domain}/u/${name}`,
'type': 'Person',
'preferredUsername': `${name}`,
'inbox': `https://${domain}/api/inbox`,
'name': displayName,
'publicKey': {
'id': `https://${domain}/u/${name}#main-key`,
'owner': `https://${domain}/u/${name}`,
'publicKeyPem': pubkey
}
};
if (imageUrl) {
actor.icon = {
'type': 'Image',
'mediaType': 'image/png',
'url': imageUrl,
};
}
return actor;
}

function createWebfinger(name, domain) {
return {
'subject': `acct:${name}@${domain}`,

'links': [
{
'rel': 'self',
'type': 'application/activity+json',
'href': `https://${domain}/u/${name}`
}
]
};
}

module.exports = router;
Oops, something went wrong.

0 comments on commit 36bb506

Please sign in to comment.
You can’t perform that action at this time.