Open source StackOverflow, Slack, Discourse, Reddit, Disqus hybrid — for your online community.
Switch branches/tags
w-km7b v0.6.15-WIP-1-3250563 v0.6.15-3250563 v0.6.14-ea9c6c9 v0.6.14-WIP-1-ea9c6c9 v0.6.13-dd15bfa v0.6.13-WIP-1-dd15bfa v0.6.12-WIP-1-3f99a2c v0.6.12-3f99a2c v0.6.11-WIP-1-7a253b5 v0.6.10-WIP-1-2f56d4c v0.6.9-WIP-3-75eef8f v0.6.9-WIP-2-ff7a125 v0.6.9-WIP-1-ad27a6e v0.6.8-WIP-2-e64f29c v0.6.8-WIP-1-eff2ca3 v0.6.7-WIP-6-0cc6df7 v0.6.7-WIP-5-3f04853 v0.6.7-WIP-4-ef6544c v0.6.7-WIP-3-381e132 v0.6.7-WIP-2-143b05f v0.6.7-WIP-1-3192d08 v0.6.6-ce61ac7 v0.6.6-WIP-2-ce61ac7 v0.6.6-WIP-1-a8c2a86 v0.6.5-eae1ceb v0.6.5-WIP-3-ad48bd0 v0.6.5-WIP-2-f2b9696 v0.6.5-WIP-1-af1a761 v0.6.4-564d7ab v0.6.3-7704eb2 v0.6.2-5d7e15e v0.6.1-WIP-3-3a1baa2 v0.6.1-WIP-2-2943dac v0.6.1-WIP-1-8ad607f v0.6.1-576508b v0.6.0-WIP-1-425770d v0.6.0-8893090 v0.5.0-WIP-4-6f60d0c v0.5.0-WIP-3-9d444b1 v0.5.0-WIP-2-f37f3be v0.5.0-WIP-1-a6b585b v0.4.8-WIP-1-42aff6e v0.4.7-f5b72f2 v0.4.6-WIP-16-c3d1a52 v0.4.6-WIP-16-7ab8b15 v0.4.6-WIP-15-aae0a12 v0.4.6-WIP-14-5f80e2a v0.4.6-WIP-13-0ea79d3 v0.4.6-WIP-12-6f32c50 v0.4.6-WIP-11-d0c5bb9 v0.4.6-WIP-10-49ab4ff v0.4.6-WIP-9-09787d2 v0.4.6-WIP-8-164ff45 v0.4.6-WIP-7-b8abb65 v0.4.6-WIP-6-61a78d2 v0.4.6-WIP-5-01d1fa2 v0.4.6-WIP-4-4b5fa65 v0.4.6-WIP-3-cfc2b6e v0.4.6-WIP-2-6a386a0 v0.4.6-WIP-1-fc0c277 v0.4.5-e6b486c v0.4.5-WIP-1-03ba618 v0.4.4-WIP-9-9892e21 v0.4.4-WIP-8-614c2a4 v0.4.4-WIP-7-fa0dd15 v0.4.4-WIP-6-31b09aa v0.4.4-WIP-5-29a3aaf v0.4.4-WIP-4-829608a v0.4.4-WIP-3-a7d8166 v0.4.4-WIP-2-d66fd1e v0.4.4-WIP-1-dd5db2a v0.4.4-55a2d15 v0.4.3-WIP-23-9b38822 v0.4.3-WIP-22-35f7e5b v0.4.3-WIP-21-599f4e8 v0.4.3-WIP-20-d10bfe1 v0.4.3-WIP-19-66c7951 v0.4.3-WIP-18-2537760 v0.4.3-WIP-17-c9bd9e3 v0.4.3-WIP-16-491830e v0.4.3-WIP-15-90dbb64 v0.4.3-WIP-12-3b65032 v0.4.3-WIP-9-ead4ca1 v0.4.3-WIP-8-39c417b v0.4.3-WIP-6-2adcd11 v0.4.3-WIP-5-2adcd11 v0.4.2-ca43efe v0.04.01 test-sourcemaps-breakpoints-wont-work test-scrypt-server-relief
Nothing to show
Clone or download
kajmagnus Remove Definitely-Typed submodule
Now using node_modules/@types/ instead.
Latest commit a040bf7 Nov 9, 2018
Permalink
Failed to load latest commit information.
app Code review, disable service worker. Typescript 3.1. Nov 9, 2018
client Code review, disable service worker. Typescript 3.1. Nov 9, 2018
conf Code review, disable service worker. Typescript 3.1. Nov 9, 2018
docker Code review, disable service worker. Typescript 3.1. Nov 9, 2018
docs Code review, disable service worker. Typescript 3.1. Nov 9, 2018
modules Remove Definitely-Typed submodule Nov 9, 2018
old Make embedded comments work again, wip. Aug 5, 2017
project VSCode config, wants Scala 2.12.4. Fix some Typescript typos. Aug 2, 2018
public Move Fontello to ty-media submodule. Add chat icons. Mar 4, 2018
s Add e2e test that repros tricky-override-group-notf-prefs issue Nov 8, 2018
tests Upgr to Typescript 3.0.3, from 2.6.1. Nov 8, 2018
to-talkyard @ d67a392 Update submodules. Oct 2, 2018
translations Add sample topics cat, for new sites Oct 30, 2018
.env Add DOCKER_REPOSITORY compose file variable. Rename proj name to 'tyd'. Oct 23, 2018
.gitignore Code review, disable service worker. Typescript 3.1. Nov 9, 2018
.gitmodules Remove Definitely-Typed submodule Nov 9, 2018
.npmignore Squash all old 4372 old commits, exclude submodules Jan 8, 2015
.yarnclean Try to prevent Yarn from deleting Selenium files, doesn't seem to work. Jan 22, 2017
COPYRIGHT.txt Upd copyright. Jun 18, 2018
LICENSE-AGPL-3.0.txt Squash all old 4372 old commits, exclude submodules Jan 8, 2015
README.md Upgr to Typescript 3.0.3, from 2.6.1. Nov 8, 2018
build.sbt Code review, disable service worker. Typescript 3.1. Nov 9, 2018
docker-compose-no-limits.yml Docker-compose 2 -> 3.1, comment out mem_limit :-(. Aug 18, 2017
docker-compose.yml Code review, disable service worker. Typescript 3.1. Nov 9, 2018
gulpfile.js Code review, disable service worker. Typescript 3.1. Nov 9, 2018
package.json Code review, disable service worker. Typescript 3.1. Nov 9, 2018
version.txt Bump version: 0.6.16 next. Oct 31, 2018
yarn.lock Code review, disable service worker. Typescript 3.1. Nov 9, 2018

README.md

Talkyard

Create a place to talk, where your users find answers to their questions, and can suggest ideas.
Place it at community.your-website.org.

  • Find answers in Question-Answers topics, like StackOverflow.
  • Gather ideas in open-ended topics, like at Reddit and Hacker News.
  • Get work done in team chat, like Slack.
  • Solve problems step by step, in flat by-time topics (coming soon).
  • Talk with your blog visitors in embedded comments, like Disqus.

Support forum here, at Talkyard.io — and report bugs there too.

Our vision is to build a tool that [people who change the world or their neighborood] can use to find answers and pick the right things to do. That's why we have Q&A (question-answers) and HackerNews & Reddit type topics, where good answers and ideas rise to the top.

Screenshots a bit below.
See it live: https://www.talkyard.io/forum/latest
Read about it, and demo forums: https://www.talkyard.io

How install?

This repository is for writing Talkyard source code, and building Docker images (docker build files are in ./docker/image-name/). To install Talkyard, instead go to: https://github.com/debiki/talkyard-prod-one ("-prod-one" means "production installation on one server").

Docker based installation. Automatic upgrades. One installation can host many sites. There's hosting, if you don't want to install it yourself.

This is beta software; there might be bugs.

Screenshots

Topic list:

ed-demo-forum-index


Question-Answers:

how-work-from-home-vpn-broken-borders


Chat:

Currently, Talkyard is a mobile friendly web app. Within half a year or a year (today is August 2018), the plan is that there'll be a white labelled mobile app. Meaning, people will be able to install your community, on their mobile phones, as a separate app with your custom icon. Push notifications for Android (however, initially not for iPhone — iPhone currently cannot do PWA mobile app push notifications).

ed-e2e-chat-owen-maria


Admin-getting-started guide:

ed-admin-intro-guide


Users online:

ed-online-users


Contributing

Want to contribute? Feel free to say hello in our community: https://www.talkyard.io/forum/. Good for you if you ask the people there what is currently being worked on, so you won't accidentally re-implement something that's almost done already — people might be working in their own work-in-progress topic branches that you don't know about.

Here's how to translate to a new language: i18n-README (step 1 and 2 only).

You need to read and agree to our Contributor License Agreement. You do that by reading it (please do — there's a human friendly intro) and appending a single line paragraph with your real name (no pseudonyms) and the following text, to all your commit messages:

I, Your Full Name <your@email.address>, agree to the Contributor License Agreement, docs/CLA-v2.txt.

Please squash your commits to just one (unless you're doing something complicated that's easier to review in separate commits).

Getting Started

Before you start

You need about 4 GB RAM for the development environment (whereas the production environment needs about 2 GB). And a somewhat fast internet connection — you'll be downloading perhaps 0.5 (?) GB Docker images.

Install Docker-Compose, version 1.7.0+: https://docs.docker.com/compose/install/, or simply, on Linux: (maybe a 'sudo' is missing on the 1st line?)

wget -qO- https://get.docker.com/ | sh
sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose
sudo chmod +x /usr/local/bin/docker-compose
docker-compose --version  # should print "docker-compose version ... build ..."

Read A brief intro to Docker-Compose — unless you know how to use docker-compose already.

The instructions

  1. Clone this repository, cd into it. Then update submodules:

    git clone https://github.com/debiki/talkyard.git talkyard
    cd talkyard
    git submodule update --init
    
  2. Append some settings to the system config so that ElasticSearch will work: (run this as one single command, not one line at a time)

    sudo tee -a /etc/sysctl.conf <<EOF
    
    ###################################################################
    # Talkyard settings
    #
    # Up the max backlog queue size (num connections per port), default = 128
    net.core.somaxconn=8192
    # ElasticSearch requires (at least) this, default = 65530
    # Docs: https://www.kernel.org/doc/Documentation/sysctl/vm.txt
    vm.max_map_count=262144
    EOF
    

    Reload the system config:

    sudo sysctl --system
    
  3. Build and start all Docker containers: (this will take a while: some Docker images will be downloaded and built)

    sudo s/d up -d   # s/d = shortcut for docker-compose, so long to type.
                     # The 's' means "scripts" and 'd' means "docker-compose".
    
    # And tail the logs:
    sudo s/d logs -f
    

    This log message might take 10 - 20 minutes: (lots of stuff is being downloaded — we'll try to include all that in the Docker image directly instead, later)

    Loading project definition from /opt/talkyard/app/project
    

    Wait until this appears in the logs:

    app_1     |
    app_1     | --- (Running the application, auto-reloading is enabled) ---
    app_1     |
    app_1     | [info] p.c.s.NettyServer - Listening for HTTP on /0:0:0:0:0:0:0:0:9000
    app_1     | [info] p.c.s.NettyServer - Listening for HTTPS on /0:0:0:0:0:0:0:0:9443
    app_1     |
    app_1     | (Server started, use Ctrl+D to stop and go back to the console...)
    app_1     |
    
  4. Compile all Scala files, start the server, as follows:

    Point your browser to http://localhost/. This sends a request to the Docker container named 'web', in which Nginx listens on port 80. Nginx sends the request to Play Framework in the 'app' container, port 9000. Play Framework then starts compiling Scala files; this takes a while — so the browser will show a 502 Bad Gateway error message (because Play didn't reply because it's busy compiling stuff).

    Eventually, when done compiling, Play Framework will start. Then this message will get logged:

    app_1  | [info] application - Starting... [EsM200HELLO]
    

    But it's easy to miss, because after that, the server logs even more messages. You can continue with the next step just below anyway — just keep reloading the browser page until any "is starting" message disappears.

  5. Create a forum

    Reload the browser at http://localhost/. Now eventually a page should be shown. Sign up as admin with this email: admin@example.com (must be that email). As username and password you can type admin and public1234.

    You'll be asked to confirm your email address, by clicking a link in an email that was sent to you — but in fact the email couldn't be sent, because you haven't configured any email server, and admin@example.com isn't your address anyway.

    Instead look at the log messages. (Run sudo docker-compose logs app if you've closed the terminal with log messages.) There you'll find the email — it's written to the log files, in development mode. Copy the confirmation link from the <a href=...> and paste it in the browser's address bar.

You can shutdown everything like so: sudo s/d-killdown, and if Play Framework runs out of memory (it'll do, if it recompiles Scala files and reloads the app many many times), you can restart it like so: sudo s/d-restart-web-app.

Troubleshooting

See tips.mnd.

Tests

End-to-end tests

The end-to-end tests are written in TypeScript and uses Selenium and Webdriver.io. See the end-to-end tests readme. And, if you want to test in a browser other than Chrome, see Making *.localhost addresses work.

Security tests

The security tests are written in TypeScript and use Tape = test-anything-protocol for Node.js. See the security tests readme.

Unit tests

Stop everything: sudo docker-compose down and then: s/cli then type test + hit Enter.

Performance tests

Install Scala SBT, see http://www.scala-sbt.org/download.html. On Linux:

echo "deb https://dl.bintray.com/sbt/debian /" | sudo tee -a /etc/apt/sources.list.d/sbt.list
sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv 2EE0EA64E40A89B84B2DF73499E82A75642AC823
sudo apt-get update
sudo apt-get install sbt

Append to /etc/security/limits.conf ... hmm but now with Docker-Compose, which container?

your_login_name hard nofile 65535
your_login_name soft nofile 65535

Configure very high max-requests-per-ip-per-second etc Nginx limits — otherwise during the performance test Nginx will start to rate limit stuff and reply 503 Service Not Available:

sudo docker-compose  -f docker-compose.yml  -f docker-compose-no-limits.yml  up -d

Technology

  • Client: React.js, TypeScript, Webdriver.io.
  • Server: Scala and Play Framework. Nginx, Nchan, some Lua. React.js in Java's Nashorn Javascript engine.
  • Databases: PostgreSQL, Redis, ElasticSearch.

Directories

This project looks like so:

server/
 |
 +-docker-compose.yml   <-- tells Docker how to run Talkyard
 |
 +-client/         <-- Javascript, CSS, React.js components
 | +-app/          <-- Client side code
 | +-server/       <-- React.js components rendered server side
 | :
 | :
 |
 +-app/            <-- Scala code — a Play Framework 2 application
 |
 +-tests/
 | +-app/          <-- Unit tests and functional tests, for the app server
 | +-e2e/          <-- End-to-end tests
 | +-security/     <-- Security tests
 |
 +-modules/
 | +-ty-dao-rdb/        <-- A database access object (DAO), for PostgreSQL
 | +-ed-core/           <-- Code shared by the DAO and by the ./app/ code
 | +-ed-prod-one-test/  <-- A production installation, for automatic tests
 | |
 | +-local/        <-- Ignored by .gitignore. Here you can override the
 | |                   default config values. If you want to, turn it into
 | |                   a Git repo.
 | |
 | ...Third party modules
 |
 +-public/         <-- Some images and libs, plus JS and CSS that Gulp
 |                     has bundled and minified from the client/ dir above.
 |
 +-docker/         <-- Dockerfiles for all docker-compose containers
 | +-web/          <-- Docker build stuff for the Nginx container
 | | +-modules/
 | |   +-nchan/    <-- WebSocket and PubSub for Nginx (a Git submodule)
 | |   +-luajit/   <-- Lua
 | |   ...
 | |
 | +-gulp/         <-- Container that runs Node.js and bundles JS and CSS
 | +-gulp-home/    <-- Mounted as Gulp container home-dir = disk cache
 | |
 | +-...           <-- More containers...
 | |
 | +-data/
 |   +-rdb         <-- Mounted as a volume in the Postgres container
 |   +-cache       <-- Mounted in the Redis container
 |   +-uploads     <-- Mounted read-write in the Play container, but
 |   |                 read-only in Nginx (to serve static files)
 |   ...
 |
 +-s/         <-- Utility scripts (typing "scripts/" is so long)
 |
 +-conf/      <-- Default config files that assume everything
                  is installed on localohost, and dev mode

Naming style, tags and a new word

CSS classes and ids

Example: s_P_By_FN-Gst. Here, s_ is a prefix used for all classes, and it means "some". For ids we use t_ instead, means "the". P means Post. By means who-was-it-written-By. FN means Full Name. Gst means Guest.

So, this is BEM (Block Element Modifier) with a few tweaks: Blocks/elements are separated with only one underscore, and modifiers with only one dash. Blocks, elems and modifiers always start with uppercase — because then it's easy to tell if we're dealing with an abbreviation or not. For example, FN (full name) is an abbreviation. But By is not (since it continues with lowercase letters).

Another example: s_Dfs_Df_Ttl — this means the title (Ttl), of a draft (Df), in a list of drafts (Dfs). You'll find abbreviations like Ttl and Df, in bem-blocks.txt.

For stuff with otherwise no class or id, and that should be clicked in end-to-end tests, we use classes only, and the prefix e_ (instead of s_ or t_).

Single and double quotes

In Typescript (and any Javascript), use single quotes for strings the computer cares about, like CSS classes or ids, e.g. className: 's_P' or reactRenderMethod = 'hydrate', or React component display names. For texts that humans read, instead use double quotes, like: Button({ ...}, "Undo"). When doing this, you can be fairly certain that if you edit a single quote string, without knowing what you're doing, something will break. Whilst if you edit a double quoted string and fix e.g. a spelling errors: the computer won't care, but humans like it.

Tag the code

Some parts of a software system, knows how other parts of the software system works, sometimes in not-obvious ways. Make such otherwise hidden duplicated knowledge visible, by tagging the code with tags like: [1ABCDE2]. Example: // Also done here: [4JKAM7] when deleting pages.. Or there's a 3rd partly lib bug workaround in one source code file, for a problem that happens in a different file, and an end-to-end test that got affected, example: [5QKBRQ]. Tag those three places with the same tag. Just type a number, random uppercase letters, and another number, to create a tag. And copy-paste it to where the related code is.

Message codes and magic underscores

Log messages, and plain text messages sent back to the browser, start with TyM if it's an info message, and TyE if it's an error. Like, "Server starting... [TyMHELLO]" (a log message).

These messsage codes helps you instantly find the relevat source code, if there's an error message anywhere. Otherwise, it can be terribly annoying, when the browser says "Not found", and you have no idea where that message comes from. For example, Nginx didn't find a location handler? Or a user is missing? Or a page? Or a post? Or a client side route is missing? Or the hostname is wrong? Or ...? And you search for "Not found" and find 1 000 matches. Now, instead, you'll see "Not found [TyE123ABC]" — and you then search for "TyE123ABC" and find the relevant source code.

Some message codes sent to the browser are checked for in end to end tests. They shall have an underscore _ at the end (because it's called end to end tests). So, if you see a message code like: "TyM0APPR_" and you change it, you need to search for it everywhere and update some end-to-end tests too.

Some message codes are checked for by production code Typescript, i.e. frontend code. They shall have a _ at the beginnign (front) of the error code, and here's how they can be used server side: throwForbidden("_TyE403BPWD", "Bad username or password") and client side: if (xhr.responseText.indexOf('_TyE403BPWD') .... — So, when you're looking at the server side code the _ tells you that the error code is used in the frontend Typescript code, so you cannot just change it.

Hen and henbirds

Source code comments should be concise, but writing "he or she" everywhere, when referring to e.g. a user, becomes a bit verbose (because "he or she" is three words). There's a short Swedish word that means "he or she", namely "hen". Let's start using it in English.

So: "hen" = either "he or she", or "him or her", depending on context. And "hens" = "his or her", and "hen's" = "he or she is".

To refer to many hen = many-he-or-she, write "people". "Hens" however means "his or her", just like "its" means, well, "its" (but not "things").

What about the bird previously called "hen"? Let's call it "henbird" instead.

So, hereafter, the word "hen" means "he or she". And the henbird, which I cannot remember having mentioned or even thought about the past year, no longer gets to occupy the short and useful word "hen".

Custom third party builds

We're building & using a smaller version of Lodash, like so: (this makes slim-bundle.min.js.gz 8kb = 4% smaller, as of September 2016)

node_modules/lodash-cli/bin/lodash  include=assign,assignIn,before,bind,chain,clone,compact,concat,create,debounce,defaults,defer,delay,each,escape,every,filter,find,findLast,flatten,flattenDeep,forEach,forOwn,has,head,includes,identity,indexOf,isArguments,isArray,isBoolean,isDate,isEmpty,isEqual,isFinite,isFunction,isNaN,isNull,isNumber,isObject,isRegExp,isString,isUndefined,iteratee,keys,last,map,matches,max,min,mixin,negate,noConflict,noop,once,pick,reduce,remove,result,size,slice,some,sortBy,sumBy,take,tap,throttle,thru,toArray,uniq,uniqBy,uniqueId,value,values \
  --output client/third-party/lodash-custom.js
  • For security reasons, we checkin only the resulting .js file (but not the .min.js) file into source control (so that you can read the source code and see what it does).
  • There are some Gulp plugins that builds Lodash but one seems abandonend (gulp-lodash-builder) and the other (gulp-lodash-custom) analyzes all .js files, I guess that'd slow down the build rather much + won't immediately work with Typescript?

Old Code

In January 2015 I squashed all old 4300+ commits into one single commit, because in the past I did some mistakes, so it feels better to start over again from commit number 1. The old commit history is available here: https://github.com/debiki/debiki-server-old

Credits

I sometimes copy ideas from Discourse, and look at its database structure, HTTP requests, and avatar pixel width. Discourse is forum software.

License

Currently AGPL — please let me know if you want me to change to GPL, contact info here: https://www.talkyard.io/contact

Copyright (c) 2010-2018  Kaj Magnus Lindberg and Debiki AB

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public License
along with this program.  If not, see <http://www.gnu.org/licenses/>.

vim: list et ts=2 sw=2 tw=0 fo=r