Launch of idoread.com

Current binary running https://idoread.com/. This production host is on Rocky 9 linux being hosted with Digital Ocean.

Currently running on a Basic Machine Type with Premium AMD 8vCPU w/ 32GB of RAM and 160GB of HDD storage. In addition to this compute instance type, a block volume of 200GB was mounted to /idoread.com-data and the stargate-tmp directory was taken from the Apario Contribution output.

This executable relies on the config.yaml file to be properly defined. If you want to know what can be defined, you can run

./idoread --help

To download the 185GB of STAR GATE files that were compiled by me, use this magnet link to download it over bittorrent.

magnet:?xt=urn:btih:5ee779797eea1c607aba772f9014b797d4c0b9c5&dn=stargate-tmp&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A6969

You can also download the .torrent file below and download it yourself. Otherwise, you can recompile it yourself from scratch.

In order to use this binary in a production environment, you can choose from these configurables:

Usage of ./idoread.macos-amd64:
  -access-log string
        Default log file for GIN access logs. (default "logs/gin.log")
  -auto-tls
        Create a self-signed certificate on the fly and use it for serving the application over SSL.
  -buffer int
        Memory allocation for CSV buffer (min 168 * 1024 = 168KB) (default 131072)
  -company-name string
        name of the company that operates the service (default "Project Apario LLC")
  -concurrent-asset-requests int
        concurrent hits to /assets/* permitted (default 369)
  -concurrent-image-views int
        concurrent hits to /covers/<doc-id>/<pg-id>/<size>.jpg permitted (default 369)
  -concurrent-pdf-downloads int
        concurrent pdf downloads permitted (default 369)
  -concurrent-searches int
        maximum number of allowed concurrent searches before a waiting room appears (default 30)
  -config string
        Configuration file (default "config.yaml")
  -cookie-domain string
        domain to use for cookies (default "localhost:8080")
  -cors-domains-csv string
        List of CORS domains in CSV format
  -csp-block-mixed-content
        Enable/Disable automatically blocking mixed HTTP and HTTPS content for requests via CSP (default true)
  -csp-child-unsafe-inline
        Enable/Disable Child SRC Unsafe Inline script execution via CSP (default true)
  -csp-domains-csv string
        List of CSP domains in CSV format
  -csp-report-uri string
        Path for content security policy violation reports to get logged (default "/security/csp-report")
  -csp-script-unsafe-eval
        Enable/Disable Unsafe Eval script execution via CSP
  -csp-script-unsafe-inline
        Enable/Disable Unsafe Inline script execution via CSP (default true)
  -csp-style-unsafe-inline
        Enable/Disable Style SRC Unsafe Inline script execution via CSP (default true)
  -csp-thirdparty-csv string
        List of third party domains in CSV format
  -csp-thirdparty-styles-csv string
        List of third party domains in CSV format
  -csp-upgrade-insecure
        Enable/Disable automagically upgrading HTTP to HTTPS for requests via CSP (default true)
  -csp-ws-domains-csv string
        List of Web Socket domains in CSV format
  -dark-mode-cookie-name string
        set the name of the cookie for dark mode (default "dark-mode")
  -database string
        apario-contribution rendered database directory path
  -decimal-symbol string
        symbol for decimals, default is . (default ",")
  -directories-limiter int
        concurrent directories to process out of the database (example: 369) (default 1)
  -directory-buffer int
        buffered channel size for pending directories from the database (3x --directories, example: 1107) (default 1)
  -enable-cors
        Enable/Disable CORS (default true)
  -enable-csp
        Enable/Disable CSP (default true)
  -error-log string
        File to write logs. (default "logs/go.log")
  -flush-database-watch-file string
        name of a file to touch in the root directory to force the app to delete the database cache and regenerate at boot (default "flush-db.next-boot")
  -limit int
        general purpose semaphore limiter (default 1)
  -load-persistent-database
        boolean to load a persisted database from disk to memory
  -log string
        File to save logs to. Default is logs/engine-YYYY-MM-DD-HH-MM-SS.log (default "logs/badbitchreads-2024-01-02-14-53-54.log")
  -pages-limiter int
        concurrent pages to process out of the database (default 1)
  -persist-runtime-database
        boolean to persist the runtime database to disk once loaded
  -persistent-database-path string
        path to runtime database file (default "database/app.db")
  -primary-domain string
        primary domain name used to access the service (default "projectapario.com")
  -rate-limit float
        Requests per second (0.5 = 1 request every 2 seconds). (default 12)
  -rate-limit-asset float
        Requests per second (0.5 = 1 request every 2 seconds). (default 36)
  -rate-limit-asset-cleanup int
        Seconds between rate limit cleanups. (default 17)
  -rate-limit-asset-ttl int
        Seconds a rate limit entry exists for before cleanup is triggered. (default 17)
  -rate-limit-cleanup int
        Seconds between rate limit cleanups. (default 3)
  -rate-limit-ttl int
        Seconds a rate limit entry exists for before cleanup is triggered. (default 3)
  -search-algorithm string
        values are wagner_fisher, ukkonen, jaro, jaro_winkler, soundex, hamming ; default is jaro_winkler (default "jaro_winkler")
  -search-concurrency-buffer int
        buffer channel size for search results ; default = 369 (default 369)
  -search-concurrency-limiter int
        concurrent keyword processing per search query ; default = 9 (default 9)
  -search-hamming-max-substitutions int
        maximum number of substitutions allowed for a word to be considered a match ; higher value = lower accuracy ; min = 1 ; default = 2 (default 2)
  -search-jaro-winkler-boost-threshold float
        weight applied to common prefixes in matched strings comparing dictionary terms, page word data, and search query params (default 0.7)
  -search-jaro-winkler-prefix-size int
        length of a jarrow weighted prefix string (default 3)
  -search-threshold-jaro float
        1.0 means exact match 0.0 means no match; default is 0.71 (default 0.71)
  -search-threshold-jaro-winkler float
        using the JaroWinkler method, define the threshold that is tolerated; default is 0.71 (default 0.71)
  -search-timeout-seconds int
        maximum seconds to spend on a search (default 30)
  -search-ukkonen-dcost int
        delete cost ; when removing a char to find a match ; increase the score by this number ; default = 1 (default 1)
  -search-ukkonen-icost int
        insert cost ; when adding a char to find a match ; increase the score by this number ; default = 1 (default 1)
  -search-ukkonen-max-substitutions int
        maximum number of substitutions allowed for a word to be considered a match ; higher value = lower accurate ; lower value = higher accuracy ; min = 0; default = 2 (default 2)
  -search-ukkonen-scost int
        substitution cost ; when replacing a char increase the score by this number ; default = 2 (default 2)
  -search-wagner-fischer-dcost int
        delete cost ; when removing a char to find a match ; increase the score by this number ; default = 1 (default 1)
  -search-wagner-fischer-icost int
        insert cost ; when adding a char to find a match ; increase the score by this number ; default = 1 (default 1)
  -search-wagner-fischer-max-substitutions int
        maximum number of substitutions allowed for a word to be considered a match ; higher value = lower accurate ; lower value = higher accuracy ; min = 0; default = 2 (default 2)
  -search-wagner-fischer-scost int
        substitution cost ; when replacing a char increase the score by this number ; default = 2 (default 2)
  -secure-port int
        Port to start the SSL version of the application. (default 8443)
  -session-store string
        where to store sessions - choices are cookie or redis ; cannot be cookie if use-cookies is false (default "cookie")
  -session-store-cookie-secret string
        a password to secure the cookies (default "secure-password-369-goes-here")
  -session-store-redis-connections int
        number of connections to maintain with redis between this application (default 10)
  -session-store-redis-database int
        the database ID in redis that sessions will be stored ; default is 3 (default 3)
  -session-store-redis-fallback-cookie
        fall back to use cookies if and when redis is temporarily unavailable
  -session-store-redis-password string
        password configured in redis that this app will use to communicate
  -session-store-redis-protocol string
        how the connection to redis is established - default is tcp (default "tcp")
  -session-store-redis-secret string
        a password to secure the redis sessions (default "secure-password-369-goes-here")
  -session-store-redis-servers string
        comma separated list of redis servers. example: '10.0.0.2:6379,10.0.0.3:6379,10.0.0.4:6379' default: 'localhost:6379' (default "localhost:6379")
  -session-store-redis-tls-certificate-path string
        where is the tls certificate for redis? (pem format required)
  -session-store-redis-tls-enabled
        is tls encryption enabled on the redis server? default is false
  -session-store-redis-tls-insecure-skip-verify
        false enforces tls certification and true disables tls verification
  -session-store-redis-tls-private-key-path string
        where is the private key for redis? (pem format required)
  -session-store-redis-tls-root-ca-path string
        where is the root ca certificate bundle for redis? (pem format required)
  -site-title string
        title of the application that appears on the web gui (default "Project Apario")
  -tls-additional-domains string
        Auto generated TLS/SSL certificates will be issued with these additional domains (CSV formatted).
  -tls-company string
        Auto generated TLS/SSL certificates are configured with the company name. (default "ACME Inc.")
  -tls-domain-name string
        Auto generated TLS/SSL certificates will have this common name and run on this domain name.
  -tls-expires-in int
        Auto generated TLS/SSL certificates will automatically expire in hours. (default 8760)
  -tls-life-min int
        Lifespan of the auto generated self signed TLS certificate in minutes. (default 72)
  -tls-private-key string
        Path to the PEM formatted SSL certificate's private key.
  -tls-private-key-password string
        If the PEM private key is encrypted with a password, provide it here.
  -tls-public-key string
        Path to the SSL certificate's public key. It expects any CA chain certificates to be concatenated at the end of this PEM formatted file.
  -tls-san-ip string
        Auto generated TLS/SSL certificates will have this SAN IP address attached to it in addition to its common name.
  -trusted-proxies string
        Configure the web server to forward client IP addresses to the application if a proxy is used such as Nginx; set that proxy's IP here.
  -unsecure-port int
        Port to start non-SSL version of application. (default 8080)
  -use-cookies
        toggle using cookies or not - cookies and sessions can be true but both cannot be false (default true)
  -use-sessions
        toggle using sessions or not - cookies and sessions can be true but both cannot be false

In addition to the config file, you'll also need to ensure that the process is running in the background.

[Unit]
Description=I Do Read Service
After=network.target

[Service]
ExecStart=/idoread.com/idoread
User=idoread
Group=idoread
Restart=always
WorkingDirectory=/idoread.com

[Install]
WantedBy=multi-user.target

This service file assumes that your primary workspace directory is /idoread.com and the database (mounted drive) is on /idoread.com-data. To install the service file:

sudo cp /idoread.com/idoread.service /etc/systemd/system/idoread.service
sudo systemctl daemon-reload
sudo systemctl enable idoread
sudo systemctl start idoread
sudo systemctl status idoread

In the /idoread.com directory, the following files are present:

╭─idoread@idoread ~/logs
╰─$ ll ..
total 94M
drwxr-x---. 12 idoread idoread 4.0K Jan  2 14:38 .
dr-xr-xr-x. 19 root    root     278 Jan  1 17:54 ..
-rw-r--r--.  1 idoread idoread 1.3K Jan  2 14:29 config.yaml
drwxr-xr-x.  3 idoread idoread   36 Jan  2 01:15 database
-rwxr-xr-x.  1 idoread idoread  94M Jan  2 14:38 idoread
-rw-r--r--.  1 idoread idoread  212 Jan  1 21:28 idoread.service
drwxr-xr-x.  2 idoread idoread   51 Jan  2 14:38 logs
drwx------.  2 idoread idoread   29 Jan  2 14:31 .ssh
drwxr-xr-x.  3 idoread idoread   69 Jan  1 21:23 .ssl

In order to define the secure-port and the unsecure-port in the configurables to values less than 1000, you'll need to run:

sudo setcap 'cap_net_bind_service=+ep' idoread

Inside the database directory will be the app.db directory which will contain a JSON dump of the maps used as basic data structures for the app. When the app boots, if this directory is populated, it'll bypass scanning the /idoread.com-data/<collection> entry for the --database configurable by loading the cached compiled output from the app.db directory instead. This gives you a boot time of a few seconds versus about 6-9 minutes. In order to utilize this functionality, it must be enabled manually in the config.yaml.

A production level of config.yaml can be seen:

---
database: "/idoread.com-data/stargate-tmp"
buffer: 3301
limit: 3301
directories-limiter: 369
pages-limiter: 3301
directory-buffer: 3301
site-title: I Do Read
enable-cors: false
enable-csp: false
auto-tls: false
tls-public-key: /idoread.com/.ssl/full-certificate.pem
tls-private-key: /idoread.com/.ssl/certificate.key
unsecure-port: 80
secure-port: 443
tls-company: "d/b/a idoread.com"
tls-domain-name: idoread.com
tls-san-ip: 127.0.0.1
tls-additional-domains: idoread.com,dev.idoread.com,local.idoread.com,www.idoread.com
csp-domains-csv: idoread.com,idoread.com:443,www.idoread.com,www.idoread.com:443
company-name: "d/b/a idoread.com"
primary-domain: idoread.com
cookie-domain: idoread.com
search-algorithm: jarow_winkler
search-threshold-jaro-winkler: 0.9
search-jaro-winkler-boost-threshold: 0.3
search-concurrency-buffer: 91604
search-concurrency-limiter: 91604
search-timeout-seconds: 30
concurrent-searches: 30
concurrent-image-views: 1107
concurrent-asset-requests: 1107
concurrent-pdf-downloads: 369
flush-database-watch-file: flush-db.next-boot
persist-runtime-database: true
load-persistent-database: true
persistent-database-path: /idoread.com/database/app.db

The associated files referenced:

╭─idoread@idoread ~
╰─$ stat /idoread.com/.ssl/full-certificate.pem
  File: /idoread.com/.ssl/full-certificate.pem -> 2024/idoread_com.full.crt
  Size: 25        	Blocks: 0          IO Block: 4096   symbolic link
Device: fc01h/64513d	Inode: 226492755   Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/ idoread)   Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:default_t:s0
Access: 2024-01-01 21:23:01.204159071 +0000
Modify: 2024-01-01 21:22:59.801122679 +0000
Change: 2024-01-01 21:22:59.801122679 +0000
 Birth: 2024-01-01 21:22:59.801122679 +0000

╭─idoread@idoread ~
╰─$ stat /idoread.com/.ssl/certificate.key
  File: /idoread.com/.ssl/certificate.key -> 2024/idoread_com.key
  Size: 20        	Blocks: 0          IO Block: 4096   symbolic link
Device: fc01h/64513d	Inode: 226492756   Links: 1
Access: (0777/lrwxrwxrwx)  Uid: ( 1000/ idoread)   Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:default_t:s0
Access: 2024-01-01 21:23:22.210703919 +0000
Modify: 2024-01-01 21:23:21.047673753 +0000
Change: 2024-01-01 21:23:21.047673753 +0000
 Birth: 2024-01-01 21:23:21.047673753 +0000

╭─idoread@idoread ~
╰─$ stat /idoread.com/database/app.db
  File: /idoread.com/database/app.db
  Size: 4096      	Blocks: 8          IO Block: 4096   directory
Device: fc01h/64513d	Inode: 218160135   Links: 2
Access: (0755/drwxr-xr-x)  Uid: ( 1000/ idoread)   Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:httpd_sys_rw_content_t:s0
Access: 2024-01-02 01:15:39.880591708 +0000
Modify: 2024-01-02 01:15:39.883591859 +0000
Change: 2024-01-02 01:15:39.883591859 +0000
 Birth: 2024-01-02 01:15:39.880591708 +0000

╭─idoread@idoread ~
╰─$ stat /idoread.com-data/stargate-tmp
  File: /idoread.com-data/stargate-tmp
  Size: 1011712   	Blocks: 2256       IO Block: 4096   directory
Device: 800h/2048d	Inode: 134217856   Links: 12341
Access: (0755/drwxr-xr-x)  Uid: ( 1000/ idoread)   Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:httpd_sys_content_t:s0
Access: 2024-01-01 18:27:45.333687550 +0000
Modify: 2023-12-10 15:42:24.000000000 +0000
Change: 2024-01-01 21:36:30.716148568 +0000
 Birth: 2024-01-01 18:27:45.333687550 +0000

FYI: This application will predictably use 16GB of RAM on its own to operate. A host with a minimum of 32GB of RAM is required. 64GB of RAM is recommended. Additionally, you'll want to ensure that you have proper storage capacities for logging. The logging directory can be its own drive mapped with /etc/fstab to give additional expandable storage to the logs drive. But that configuration topology is not currently implemented.

In addition to this general configuration, this production service is running with SELinux set to enforcing. In order to make this compatible, effective chcon commands need to be applied.

sudo chcon -R -t httpd_sys_content_t /idoread.com-data
sudo chcon -R -t httpd_sys_rw_content_t /idoread.com/database
sudo chcon -R -t httpd_sys_rw_content_t /idoread.com/logs

╭─idoread@idoread ~
╰─$ sudo getenforce
Enforcing