Serval App

[johnml1135]

Create a simple python application that can build NMT jobs and return pretranslations. This app may be extended to be a bit more, but will always be a prototype/sample app.

Functional Requirements for MVP:

• Users receive a passcode from our team to be able to use the service
• Users are given a fair amount of instruction as to what is going on
• Users (without authentication beyond the passcode), enter:
  • An email address to send (1) confirmation of job started (2) an error if it errors out somehow and (3) the resultant translations
  • 2 paratext projects for training, one source and one target
  • 1 or 2 paratext projects for translating (source needed, target optional)
  • Language codes for the source and target languages
  • Press a button to start and then tell them to check their email

Other Requirements for MVP:

• Similar to https://translator.multilingualai.com/
  • Uses https://streamlit.io/ (or is there a better one?)
• Use production Serval as backend - https://prod.serval-api.org/swagger/index.html
• We provide a passcode so that people can use it and share it in their groups, but that the internet in general does not have access
• Talk to Bryan Vold about getting an email to use.

Challenges post MVP:

• Instead of getting back json or text, can we recreate a paratext project? Can we steal code from https://github.com/sillsdev/silnlp to do so?

[Enkidu93]

This looks very doable and right up my alley. (I miss python 😛). I imagine this would be best started in a fresh repo (???).

[johnml1135]

@ddaspit - a subfolder in Serval? It could be it's own repo then if it had it's own CI, etc., it could work easier.

[ddaspit]

From a development perspective, the main benefit of using the Serval repo is that API changes and the corresponding changes to the web app could be made in the same PR. It would be easier to keep Serval and the web app in sync. If the web app is built on a different tech stack (Python, streamlit, etc.), it can be tricky combine the build systems. I usually separate codebases if they are built on different languages and tech stacks. Machine is split into 3 repos: Machine.NET, Machine.py, and Machine.js. What about deployment? @johnml1135, how would separate repos affect deployment? I assume that the Serval web app would be deployed in our Kubernetes cluster.

[johnml1135]

The biggest concern would be separating the CI with both - but it actually can be done: https://stackoverflow.com/questions/63822219/how-to-run-github-actions-workflow-only-if-the-pushed-files-are-in-a-specific-fo. Therefore, I recommend the mono-repo - add python to the serval repo.

We can use streamlit for hosting the app - it should be free, because we will be publicly hosting the app:

It appears that it can be auto-deployed from a GitHub repo: https://docs.streamlit.io/streamlit-community-cloud/deploy-your-app

[Enkidu93]

The biggest concern would be separating the CI with both - but it actually can be done: https://stackoverflow.com/questions/63822219/how-to-run-github-actions-workflow-only-if-the-pushed-files-are-in-a-specific-fo. Therefore, I recommend the mono-repo - add python to the serval repo.

We can use streamlit for hosting the app - it should be free, because we will be publicly hosting the app:

It appears that it can be auto-deployed from a GitHub repo: https://docs.streamlit.io/streamlit-community-cloud/deploy-your-app

Sounds good 👍 . And yes, the CI shouldn't be an issue.

[ddaspit]

I'm fine with a monorepo approach.

[Enkidu93]

@johnml1135 @ddaspit See here my initial work - an absolute MVP.

The UI:

It sends emails to the address associated with the build when the build starts, faults or is canceled, and when it is finished - at which point it attaches a text file with the pretranslations that looks something like the following:

...

source.txt:verse_001_001	The book of the birth of Jesus Christ , the son of David , the son of Abraham .
source.txt:verse_001_002	Abraham was the father of Isaac, Isaac the father of Jacob, and Jacob the father of Judas and his brothers.
source.txt:verse_001_003	Judah begat Pharaoh and Sarah in Tamar; and Pharaoh begat Shoram; and Shoram begat Ram;
source.txt:verse_001_004	and Miriam the father of Minadab, and Minadab the father of Nahason, and Nahason the father of Salmon,
source.txt:verse_001_005	and Salmon begot Boaz of Ahab; and Boaz begot David of Judah; and David begot Jesse;
source.txt:verse_001_006	And in the same year was born David the king .

...

I'm sure I'll be jumping ship for a bit to help push things out for this weekend, but if you have thoughts, let me know.

[johnml1135]

Great - is it hosted on streamlit yet?

[johnml1135]

Instead of the passcode, use Serval Auth0 creds - and hard-code the production URL and Auth0 Token URL.