Project details

This is a free, open source web app, developed by the Open Philology project at Leiden University, the aim of which is to assist scholars in the work of producing scholarly editions of premodern texts, mainly those in Tibetan and Chinese, but with applicability beyond these two linguistic registers.

The general aims of the project:

• provide tools for editing multilingual texts,
• allow for easy export of edited text into standard word processors/typesetters for further/final typesetting (for paper publication),
• allow online display of editions, both as completed projects and in process.

Useful links:

• Frontend Application

Technology stack

• Ruby v3.2.1
• Ruby on Rails v7.0.6
• Redis
• PostgreSQL

Project setup instructions

Project should be setup from staging branch.

Note: M1 users should use -arch x86_64 flag for any failing installations

1. Create local .env files

• cp .env.local.template .env.development.local
• cp .env.local.template .env.test.local

2. Set up ENV variables in .env.development.local and .env.test.local

• update DATABASE_URL with your database username and password
• for SECRET_KEY_BASE and DEVISE_JWT_SECRET_KEY use two values generated by rails secret

3. Install Bundler and gems

• gem install bundler
• bundle install

4. Instal Lefthook

• lefthook install -f

If you don't run the above, you won't be able to run tests and use left hook.

5. Prepare local databases

• rails db:setup

6. Start Rails server & Sidekiq background job processor

• rails server
• bundle exec sidekiq -C config/sidekiq.yml

Application is running on http://localhost:3000

Running tests

• rspec spec to run tests.

Testing CI locally

Requirements

• docker
• act

Commands

• act -P ubuntu-latest=lucasalt/act_base:latest to run the actions locally

API Docs

Rswag is used to generate API docs semi-automatically.

Documentation is generated in OpenAPI 3.0.1 interface based on integration specs built for controllers.

Please use rails generate rspec:swagger V1::ControllerName to generate a plain spec file for the controller. Please enclose specs used to define documentation files in a context with swagger: true tag, eg: context 'swagger docs generation', swagger: true do ... end. These will be automatically excluded from "normal" specs. Only tests tagged as swagger: true will be used to generate docs.

Once tests are ready, use RAILS_ENV=test rails rswag command to generate the JSON file, and /api-docs endpoint to preview the documentation in swagger UI.

Production deployment

We recommend using Dokku for building and managing the staging & production application.

The following ENV variables are required in production environment:

• APP_HOST Rails application host
• DATABASE_URL with the details of production database
• DEFAULT_SENDER will be used as a default sender of all the notifications
• DEVISE_JWT_SECRET_KEY each environment should have unique key generated by rails secret
• ENVIRONMENT_NAME used to differentiate environments in Sentry (for example, staging and production)
• FRONTEND_APP_CONFIRM_ACCOUNT_URL the URL to the confirm account page in FE app (<FE app URL>/confirm-account)
• FRONTEND_APP_ORIGINS allowed FE origins (for CORS)
• FRONTEND_APP_RESET_PASSWORD_URL the URL to the reset password form in FE app (<FE app URL>/new-password)
• FRONTEND_APP_SIGN_IN_URL the URL to the login form in FE app (<FE app URL>/sign-in)
• FRONTEND_APP_USERS_MANAGEMENT_URL the URL to users management page in FE app (<FE app URL>/manage-users)
• RACK_ENV set to production
• RAILS_ENV set to production
• RAILS_SERVE_STATIC_FILES set to enabled
• REDIS_URL URL to Redis instance
• SECRET_KEY_BASE each environment should have unique key generated by rails secret
• SENDGRID_USER_NAME and SENDGRID_PASSWORD with Sendgrid api key
• SENTRY_DSN for Sentry config
• SIGNUP_NOTIFICATION_RECIPIENT an email address that should receive notifications about new signups

User Instructions

This is a simple introduction to OPEn for first time users. It will help you get started with your first editing Project.

The Introduction does not get into any detail about the editing process itself: once you have started your Project, you fill find that the different functionalities within the app are on the whole self-explanatory. The challenge is to get a new Project going, and this is what this Instruction will help you do.

OPEn was designed to be compatible with CollateX output files. In principle, while it is possible to start with a file in any format (including plain text), the key functionalities of OPEn will work only if the file is ‘tokenized’ ideally in .json format.

If you have a single witness of your text to start with, perhaps in a plain text format, find a way to tokenize that file and convert it into .json.

If you have multiple witness files, we recommend that you collate them using the CollateX script.

Pre-processing

But before you run CollateX on your files, we strongly suggest that you pre-process your files by cleaning them up of unwanted signs and by standardizing their format. This will greatly facilitate your editing work later on.

Eventually we would like to write a script to do this pre-processing automatically, but for the time being it is not too difficult to do it by hand.

Below we outline the procedure that we like for p there-processing of witnesses of Tibetan texts in romanized Wylie transliteration. (For e.g. Chinese, the procedure will be slightly different).

The guidelines below include specific instructions for the use of the ‘replace’ function with regular expressions (RegEx). For this, you can use any program whose ‘replace’ functionalities allow the application of RegEx. There are many such programs available.

In each of your witness files individually:

1. Replace backslashes with vertical vars
2. Replace double shads with space in between with double shads with no space in between
3. Add space after last letter and before single shad

   ([a-z])(\|)
   \1 \2
   

4. Add space after shad and before first letter

   (\|)([a-z])
   \1 \2
   

5. Add space after shad and before a’chung

   (\|)(') 
   \1 \2
   

Run CollateX

Once you have pre-processed your witness files, it’s time to run CollateX on them.

Download CollateX from https://collatex.net/

Run the collatex script with the following arguments:

-f json
-t

The basic command will look like this:

java -jar collatex-tools-1.7.1.jar -f json -t -o [outputfilename.json] [file1.txt] [file2.txt] [file3.txt]…

Ideally you want to keep your input witness files in the same folder in which you have the CollateX script.

Depending on the size and number of your input witness files, the CollateX operation can take a long time, so be patient!

The output file (in this case, outputfilename.json) is the file you will then feed into OPEn when you 'Start a New Project.'

Add witness sigla to json file

But before you can import the json file into OPEn to start your New Project, you need to do open the json file in any text editing program (e.g. Notepad) and manually insert the witness sigla corresponding to the input witness files that you collated in CollateX.

For example, if in the CollateX operation above file1.txt contained the text of the Basgo, the corresponding siglum will be D. If file2.txt was the Derge woodblock print, the siglum will be D. And so on. For sigla of Tibetan canonical collections, we follow the guidelines recommended by Paul Harrison and Helmut Eimer, "Kanjur and Tanjur Sigla: A Proposal for Standardization," in Transmission of the Tibetan Canon: Papers Presented at a Panel of the 7th Seminar of the International Association for Tibetan Studies, Graz 1995, edited by Helmut Eimer (Verlag der Österreichischen Akademie der Wissenschaften, 1997).

The result you are aiming at is a .json file that begins with:

{"witnesses":["Siglum for file1.txt","Siglum for file2.txt","Siglum for file3.txt"…],"table":…