RESTful API for converting clinical documents and files
Docsmith is a RESTful API, built using Node.js and the Fastify web framework, that can convert files from:
- DOCX to HTML
- DOCX to TXT
- PDF to HTML
- PDF to TXT
- RTF to HTML
- RTF to TXT
- Scanned documents (as PDFs) to TXT using OCR
Docsmith was created out of a need for an open-source document conversion service at Yeovil District Hospital NHS Foundation Trust.
Being open-source, with the ability to be self-hosted, enables a data processor (i.e an NHS trust) to confirm that a service is not storing and logging files with confidential patient identifiable data (PID) in them, which is essential for preventing potential GDPR breaches. This is something that the majority of existing closed-source document conversion services cannot offer. Docsmith was built to remedy this.
Before Docsmith, Yeovil District Hospital was using expensive black-box conversion tools that would regularly produce unreadable documents with issues such as text running off the page, paragraphs overlapping each other, and Windows-1252 to UTF-8 character encoding problems. GP surgeries in Somerset and Dorset would receive these corrupted documents through MESH and be unable to read them. This resulted in time and money wasted either posting or faxing them again; opening up the potential for further data breaches.
Docsmith enables a data processor to use a robust, GDPR-compliant, open-source document conversion service. In comparison with equivalents in the market today it completes this vital task at a fraction of the cost (free!), whilst also ensuring a higher level of security and privacy for the data subjects.
- Node.js
- Linux only: latest available
poppler-data
andpoppler-utils
binaries - Linux and macOS only: latest available
unrtf
binary
Perform the following steps before deployment:
- Clone the repo
- Navigate to the project directory
- Run
npm install --ignore-scripts --production
to install dependencies - Make a copy of
.env.template
in the root directory and rename it to.env
- Configure the application using the environment variables in
.env
- Place additional trained data into
ocr_lang_data
directory (optional, info can be found here)
Note: Set the following environment variables in .env
to meet NHS Digital's recommendation to retain 6 months' worth of logs:
LOG_ROTATION_DATE_FORMAT="YYYY-MM-DD"
LOG_ROTATION_FREQUENCY="daily"
LOG_ROTATION_MAX_LOGS="180"
- Run
npm start
The service should be up and running on the port set in the config. You should see the following output in stdout or the log file specified using the LOG_ROTATION_FILENAME
environment variable:
{
"level": "info",
"time": "2020-12-01T09:48:08.612Z",
"pid": 41896,
"hostname": "MYCOMPUTER",
"msg": "Server listening at http://0.0.0.0:8204"
}
You can now navigate to http://0.0.0.0:8204/docs to see the API documentation!
This requires Docker installed.
- Run
docker compose up
(ordocker compose up -d
to run in background)
If you are unable to deploy this into production using Docker, it is recommended that you use a process manager such as PM2.
- Run
npm install -g pm2
to install pm2 globally - Launch application with
pm2 start .pm2.config.js
- Check the application has been deployed using
pm2 list
orpm2 monit
If using a Microsoft Windows OS utilise pm2-installer to install PM2 as a Windows service.
Note: PM2 will automatically restart the application if .env
is modified.
Contributions are welcome, and any help is greatly appreciated!
See CONTRIBUTING.md for details on how to get started. Please adhere to this project's Code of Conduct when contributing.
docsmith
is licensed under the MIT license.