Current binary running https://idoread.com/. This production host is on Rocky 9 linux being hosted with Digital Ocean.
Currently running on a Basic Machine Type with Premium AMD 8vCPU w/ 32GB of RAM and 160GB of HDD storage. In addition to this compute instance type, a block volume of 200GB was mounted to /idoread.com-data and the stargate-tmp directory was taken from the Apario Contribution output.
This executable relies on the config.yaml file to be properly defined. If you want to know what can be defined, you can run
./idoread --helpTo download the 185GB of STAR GATE files that were compiled by me, use this magnet link to download it over bittorrent.
magnet:?xt=urn:btih:5ee779797eea1c607aba772f9014b797d4c0b9c5&dn=stargate-tmp&tr=udp%3A%2F%2Ftracker.openbittorrent.com%3A6969
You can also download the .torrent file below and download it yourself. Otherwise, you can recompile it yourself from scratch.
In order to use this binary in a production environment, you can choose from these configurables:
Usage of ./idoread.macos-amd64:
-access-log string
Default log file for GIN access logs. (default "logs/apario-reader-gin-2024-04-22-03-16-22.log")
-ads-txt-path string
Relative path to override the /ads.txt entry that helps fight fraud.
-auth-max-failed-logins int
maximum failed logins before the account is locked (default 17)
-auto-tls
Create a self-signed certificate on the fly and use it for serving the application over SSL.
-buffer int
Memory allocation for CSV buffer (min 168 * 1024 = 168KB) (default 131072)
-cache-control-assets-seconds int
seconds for http header Cache-Control max-age=3600 [default 1 hour] (default 3600)
-cache-control-database-seconds int
seconds for http header Cache-Control max-age=1209600 [default 14 days] (default 1209600)
-company-name string
name of the company that operates the service (default "Project Apario LLC")
-concurrent-asset-requests int
concurrent hits to /assets/* permitted (default 369)
-concurrent-image-views int
concurrent hits to /covers/<doc-id>/<pg-id>/<size>.jpg permitted (default 369)
-concurrent-pdf-downloads int
concurrent pdf downloads permitted (default 369)
-concurrent-searches int
maximum number of allowed concurrent searches before a waiting room appears (default 30)
-config string
Configuration file (default "config.yaml")
-cookie-domain string
domain to use for cookies (default "localhost:8080")
-cors-allow-credentials
Define the header value for Access-Control-Allow-Credentials
-cors-allow-headers string
Define the header value for Access-Control-Allow-Headers (default "Origin, Content-Type, Content-Length, Accept-Encoding, X-CSRF-Token, Authorization")
-cors-allow-methods string
Define the header value for Access-Control-Allow-Methods (default "GET, POST, PUT, DELETE, OPTIONS")
-cors-allow-origin string
Define the header value for Access-Control-Allow-Origin (default "*")
-csp-block-mixed-content
Enable/Disable automatically blocking mixed HTTP and HTTPS content for requests via CSP (default true)
-csp-child-unsafe-inline
Enable/Disable Child SRC Unsafe Inline script execution via CSP (default true)
-csp-domains-csv string
List of CSP domains in CSV format
-csp-report-uri string
Path for content security policy violation reports to get logged (default "/security/csp-report")
-csp-script-unsafe-eval
Enable/Disable Unsafe Eval script execution via CSP
-csp-script-unsafe-inline
Enable/Disable Unsafe Inline script execution via CSP (default true)
-csp-style-unsafe-inline
Enable/Disable Style SRC Unsafe Inline script execution via CSP (default true)
-csp-thirdparty-csv string
List of third party domains in CSV format
-csp-thirdparty-styles-csv string
List of third party domains in CSV format
-csp-upgrade-insecure
Enable/Disable automagically upgrading HTTP to HTTPS for requests via CSP (default true)
-csp-ws-domains-csv string
List of Web Socket domains in CSV format
-dark-mode-cookie-name string
set the name of the cookie for dark mode (default "dark-mode")
-database string
apario-contribution rendered database directory path
-database-concurrent-write-semaphore int
concurrent disk write operations permitted to acquire a lock (default 17000)
-decimal-symbol string
symbol for decimals, default is . (default ",")
-directories-limiter int
concurrent directories to process out of the database (example: 369) (default 1)
-directory-buffer int
buffered channel size for pending directories from the database (3x --directories, example: 1107) (default 1)
-enable-ads-txt
Enable the endpoint for /ads.txt to be served. Required to use --ads-txt-path.
-enable-cors
Enable/Disable CORS (default true)
-enable-csp
Enable/Disable CSP (default true)
-enable-middleware-asset-rate-limiting
Toggle the tollbooth rate limiter for asset routes. (default true)
-enable-middleware-download-rate-limiting
Toggle the tollbooth rate limiter for downloads routes. (default true)
-enable-middleware-ip-ban-list
Enable the middleware for ip ban list. (default true)
-enable-middleware-rate-limiting
Toggle the tollbooth rate limiter for normal routes (default true)
-enable-middleware-tls-handshake-check
Toggle whether to return an error on misconfigured TLS requests (default true)
-enable-ping
Enable the /ping endpoint of your application to return PONG. (default true)
-enable-security-txt
Enable the endpoint for /security.txt to be served. Required to use --security-txt-path.
-environment string
environment label (default "development")
-flush-database-watch-file string
name of a file to touch in the root directory to force the app to delete the database cache and regenerate at boot (default "flush-db.next-boot")
-force-https
force-https when true will redirect any request into --unsecure-port to --secure-port using middleware
-gin-log-stdout
send gin logs to stdout (default true)
-hits-file string
File that contains a JSON encoded value of the hits on the site. (default "database/hits.db")
-identifier-year-end-max int
if > 0 then time.Now().Year() will be compared against this as a maximum cutoff year to accept a document record
-identifier-year-offset int
+/- years from current year to look for documents in the database from when they were generated (default 17)
-identifier-year-start-min int
if > 0 then time.Now().Year() will be compared against this as a minimum cutoff to accept a document record
-info-log string
File to save logs to. Default is logs/engine-YYYY-MM-DD-HH-MM-SS.log (default "logs/apario-reader-info-2024-04-22-03-16-22.log")
-ip-ban-file string
File that contains JSON encoded values for the IP Ban list. (default "database/ip.db")
-ip-ban-list-sync-delay int
seconds between synchronizing the ip ban list to disk (default 3600)
-limit int
general purpose semaphore limiter (default 1)
-load-persistent-database
boolean to load a persisted database from disk to memory
-no-route-path-contains-watch-list string
Pipe separated string of partial routes that should trigger an IP ban if too many are received.
-no-route-path-watch-list string
Pipe separated string of routes that should trigger an IP ban if too many are received.
-online-refresh-delay-minutes int
seconds to count active online users before offline cut-off. 369 seconds = 6 minutes 9 seconds = default (default 17)
-pages-limiter int
concurrent pages to process out of the database (default 1)
-persist-runtime-database
boolean to persist the runtime database to disk once loaded
-persistent-database-path string
path to runtime database file (default "database/app.db")
-primary-domain string
primary domain name used to access the service (default "projectapario.com")
-product-name string
name of the product that is running, used as the prefix to the log file and throughout the runtime of the app; this is its self label (default "apario-reader")
-production-environment-label string
default is production but useful when --environment value must be treated like production without using the label production. if you dont know how to use this flag, dont use it. (default "production")
-rate-limit float
Requests per second (0.5 = 1 request every 2 seconds). (default 12)
-rate-limit-asset float
Requests per second (0.5 = 1 request every 2 seconds). (default 36)
-rate-limit-asset-cleanup int
Seconds between rate limit cleanups. (default 17)
-rate-limit-asset-ttl int
Seconds a rate limit entry exists for before cleanup is triggered. (default 17)
-rate-limit-cleanup int
Seconds between rate limit cleanups. (default 3)
-rate-limit-download float
Requests per second (0.5 = 1 request every 2 seconds). (default 36)
-rate-limit-download-cleanup int
Seconds between rate limit cleanups. (default 17)
-rate-limit-download-ttl int
Seconds a rate limit entry exists for before cleanup is triggered. (default 17)
-rate-limit-ttl int
Seconds a rate limit entry exists for before cleanup is triggered. (default 3)
-robots-txt-path string
Relative path to override the /robots.txt entry that denies all crawlers.
-search-algorithm string
values are wagner_fisher, ukkonen, jaro, jaro_winkler, soundex, hamming ; default is jaro_winkler (default "jaro_winkler")
-search-concurrency-buffer int
buffer channel size for search results ; default = 369 (default 369)
-search-concurrency-limiter int
concurrent keyword processing per search query ; default = 9 (default 9)
-search-hamming-max-substitutions int
maximum number of substitutions allowed for a word to be considered a match ; higher value = lower accuracy ; min = 1 ; default = 2 (default 2)
-search-jaro-winkler-boost-threshold float
weight applied to common prefixes in matched strings comparing dictionary terms, page word data, and search query params (default 0.7)
-search-jaro-winkler-prefix-size int
length of a jarrow weighted prefix string (default 3)
-search-threshold-jaro float
1.0 means exact match 0.0 means no match; default is 0.71 (default 0.71)
-search-threshold-jaro-winkler float
using the JaroWinkler method, define the threshold that is tolerated; default is 0.71 (default 0.71)
-search-timeout-seconds int
maximum seconds to spend on a search (default 30)
-search-ukkonen-dcost int
delete cost ; when removing a char to find a match ; increase the score by this number ; default = 1 (default 1)
-search-ukkonen-icost int
insert cost ; when adding a char to find a match ; increase the score by this number ; default = 1 (default 1)
-search-ukkonen-max-substitutions int
maximum number of substitutions allowed for a word to be considered a match ; higher value = lower accurate ; lower value = higher accuracy ; min = 0; default = 2 (default 2)
-search-ukkonen-scost int
substitution cost ; when replacing a char increase the score by this number ; default = 2 (default 2)
-search-wagner-fischer-dcost int
delete cost ; when removing a char to find a match ; increase the score by this number ; default = 1 (default 1)
-search-wagner-fischer-icost int
insert cost ; when adding a char to find a match ; increase the score by this number ; default = 1 (default 1)
-search-wagner-fischer-max-substitutions int
maximum number of substitutions allowed for a word to be considered a match ; higher value = lower accurate ; lower value = higher accuracy ; min = 0; default = 2 (default 2)
-search-wagner-fischer-scost int
substitution cost ; when replacing a char increase the score by this number ; default = 2 (default 2)
-secure-port int
Port to start the SSL version of the application. (default 8443)
-security-txt-path string
Relative path to override the /security.txt entry that helps fight fraud.
-session-authenticity-token-secret string
secret for session-authenticity-token that is used for aes-gcm encryption
-session-concurrent-crypt-actions-limit int
concurrent encrypt/decrypt calls permitted. change this value if performance is being impacted by excessive authentication requests. (default 1776)
-session-secret string
secret key used for securing sessions
-sessions-directory string
absolute path of a directory that sessions can be stored
-site-title string
title of the application that appears on the web gui (default "Project Apario")
-snippets-database-path string
absolute path to the snippets.db file for persistent snippets (default "database/snippets.db")
-tag-database-path string
absolute path to the tags.db file for persistent storage (default "database/tags.db")
-textee-database string
absolute path to textee database directory (default "database/textee.db")
-tls-additional-domains string
Auto generated TLS/SSL certificates will be issued with these additional domains (CSV formatted).
-tls-company string
Auto generated TLS/SSL certificates are configured with the company name. (default "ACME Inc.")
-tls-domain-name string
Auto generated TLS/SSL certificates will have this common name and run on this domain name.
-tls-expires-in int
Auto generated TLS/SSL certificates will automatically expire in hours. (default 8760)
-tls-life-min int
Lifespan of the auto generated self signed TLS certificate in minutes. (default 72)
-tls-private-key string
Path to the PEM formatted SSL certificate's private key.
-tls-private-key-password string
If the PEM private key is encrypted with a password, provide it here.
-tls-public-key string
Path to the SSL certificate's public key. It expects any CA chain certificates to be concatenated at the end of this PEM formatted file.
-tls-san-ip string
Auto generated TLS/SSL certificates will have this SAN IP address attached to it in addition to its common name.
-trusted-proxies string
Configure the web server to forward client IP addresses to the application if a proxy is used such as Nginx; set that proxy's IP here.
-unsecure-port int
Port to start non-SSL version of application. (default 8080)
-use-cookies
toggle using cookies or not - cookies and sessions can be true but both cannot be false (default true)
-use-sessions
toggle using sessions or not - cookies and sessions can be true but both cannot be false
-user_identifier_length int
char length of the user identifier ; default is 6 which gives you 6^36 = 1.50094635e17 possibilities ; thats a lot! (default 6)
-users-database-path string
absolute path to store user directory information (default "database/user.db")
In addition to the config file, you'll also need to ensure that the process is running in the background.
[Unit]
Description=I Do Read Service
After=network.target
[Service]
ExecStart=/idoread.com/idoread
User=idoread
Group=idoread
Restart=always
WorkingDirectory=/idoread.com
[Install]
WantedBy=multi-user.targetThis service file assumes that your primary workspace directory is /idoread.com and the database (mounted drive) is on /idoread.com-data. To install the service file:
sudo cp /idoread.com/idoread.service /etc/systemd/system/idoread.service
sudo systemctl daemon-reload
sudo systemctl enable idoread
sudo systemctl start idoread
sudo systemctl status idoreadIn the /idoread.com directory, the following files are present:
╭─idoread@idoread ~/logs
╰─$ ll ..
total 94M
drwxr-x---. 12 idoread idoread 4.0K Jan 2 14:38 .
dr-xr-xr-x. 19 root root 278 Jan 1 17:54 ..
-rw-r--r--. 1 idoread idoread 1.3K Jan 2 14:29 config.yaml
drwxr-xr-x. 3 idoread idoread 36 Jan 2 01:15 database
-rwxr-xr-x. 1 idoread idoread 94M Jan 2 14:38 idoread
-rw-r--r--. 1 idoread idoread 212 Jan 1 21:28 idoread.service
drwxr-xr-x. 2 idoread idoread 51 Jan 2 14:38 logs
drwx------. 2 idoread idoread 29 Jan 2 14:31 .ssh
drwxr-xr-x. 3 idoread idoread 69 Jan 1 21:23 .ssl
In order to define the secure-port and the unsecure-port in the configurables to values less than 1000, you'll need to run:
sudo setcap 'cap_net_bind_service=+ep' idoreadInside the database directory will be the app.db directory which will contain a JSON dump of the maps used as basic data structures for the app. When the app boots, if this directory is populated, it'll bypass scanning the /idoread.com-data/<collection> entry for the --database configurable by loading the cached compiled output from the app.db directory instead. This gives you a boot time of a few seconds versus about 6-9 minutes. In order to utilize this functionality, it must be enabled manually in the config.yaml.
A production level of config.yaml can be seen:
---
database: "/idoread.com-data/stargate-tmp"
buffer: 3301
limit: 3301
directories-limiter: 369
pages-limiter: 3301
directory-buffer: 3301
site-title: I Do Read
enable-cors: false
enable-csp: false
auto-tls: false
tls-public-key: /idoread.com/.ssl/full-certificate.pem
tls-private-key: /idoread.com/.ssl/certificate.key
unsecure-port: 80
secure-port: 443
tls-company: "d/b/a idoread.com"
tls-domain-name: idoread.com
tls-san-ip: 127.0.0.1
tls-additional-domains: idoread.com,dev.idoread.com,local.idoread.com,www.idoread.com
csp-domains-csv: idoread.com,idoread.com:443,www.idoread.com,www.idoread.com:443
company-name: "d/b/a idoread.com"
primary-domain: idoread.com
cookie-domain: idoread.com
search-algorithm: jarow_winkler
search-threshold-jaro-winkler: 0.9
search-jaro-winkler-boost-threshold: 0.3
search-concurrency-buffer: 91604
search-concurrency-limiter: 91604
search-timeout-seconds: 30
concurrent-searches: 30
concurrent-image-views: 1107
concurrent-asset-requests: 1107
concurrent-pdf-downloads: 369
flush-database-watch-file: flush-db.next-boot
persist-runtime-database: true
load-persistent-database: true
persistent-database-path: /idoread.com/database/app.db
no-route-path-watch-list: "/wp-login.php"
no-route-path-contains-watch-list: "/wp-includes|/cgi-bin|/.htpasswd|/.htaccess"The associated files referenced:
╭─idoread@idoread ~
╰─$ stat /idoread.com/.ssl/full-certificate.pem
File: /idoread.com/.ssl/full-certificate.pem -> 2024/idoread_com.full.crt
Size: 25 Blocks: 0 IO Block: 4096 symbolic link
Device: fc01h/64513d Inode: 226492755 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ idoread) Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:default_t:s0
Access: 2024-01-01 21:23:01.204159071 +0000
Modify: 2024-01-01 21:22:59.801122679 +0000
Change: 2024-01-01 21:22:59.801122679 +0000
Birth: 2024-01-01 21:22:59.801122679 +0000
╭─idoread@idoread ~
╰─$ stat /idoread.com/.ssl/certificate.key
File: /idoread.com/.ssl/certificate.key -> 2024/idoread_com.key
Size: 20 Blocks: 0 IO Block: 4096 symbolic link
Device: fc01h/64513d Inode: 226492756 Links: 1
Access: (0777/lrwxrwxrwx) Uid: ( 1000/ idoread) Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:default_t:s0
Access: 2024-01-01 21:23:22.210703919 +0000
Modify: 2024-01-01 21:23:21.047673753 +0000
Change: 2024-01-01 21:23:21.047673753 +0000
Birth: 2024-01-01 21:23:21.047673753 +0000
╭─idoread@idoread ~
╰─$ stat /idoread.com/database/app.db
File: /idoread.com/database/app.db
Size: 4096 Blocks: 8 IO Block: 4096 directory
Device: fc01h/64513d Inode: 218160135 Links: 2
Access: (0755/drwxr-xr-x) Uid: ( 1000/ idoread) Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:httpd_sys_rw_content_t:s0
Access: 2024-01-02 01:15:39.880591708 +0000
Modify: 2024-01-02 01:15:39.883591859 +0000
Change: 2024-01-02 01:15:39.883591859 +0000
Birth: 2024-01-02 01:15:39.880591708 +0000
╭─idoread@idoread ~
╰─$ stat /idoread.com-data/stargate-tmp
File: /idoread.com-data/stargate-tmp
Size: 1011712 Blocks: 2256 IO Block: 4096 directory
Device: 800h/2048d Inode: 134217856 Links: 12341
Access: (0755/drwxr-xr-x) Uid: ( 1000/ idoread) Gid: ( 1000/ idoread)
Context: unconfined_u:object_r:httpd_sys_content_t:s0
Access: 2024-01-01 18:27:45.333687550 +0000
Modify: 2023-12-10 15:42:24.000000000 +0000
Change: 2024-01-01 21:36:30.716148568 +0000
Birth: 2024-01-01 18:27:45.333687550 +0000
FYI: This application will predictably use 16GB of RAM on its own to operate. A host with a minimum of 32GB of RAM is required. 64GB of RAM is recommended. Additionally, you'll want to ensure that you have proper storage capacities for logging. The logging directory can be its own drive mapped with /etc/fstab to give additional expandable storage to the logs drive. But that configuration topology is not currently implemented.
In addition to this general configuration, this production service is running with SELinux set to enforcing. In order to make this compatible, effective chcon commands need to be applied.
sudo chcon -R -t httpd_sys_content_t /idoread.com-data
sudo chcon -R -t httpd_sys_rw_content_t /idoread.com/database
sudo chcon -R -t httpd_sys_rw_content_t /idoread.com/logs╭─idoread@idoread ~
╰─$ sudo getenforce
Enforcing