This service is used to help people find the correct Harmonised System (HS) code, duties, rules of origin etc for the products that they want to export to the UK.
- Python 3
- Node Active LTS version (Current Active version is v14)
- Directory Forms API (https://github.com/uktrade/directory-forms-api)
- redis (installed locally)
- postgres (installed locally)
If you have Docker installed, you can run this service without needing to set up the database yourself or worrying about virtual environments - it's all within the Docker instance.
- Docker for mac - https://hub.docker.com/editions/community/docker-ce-desktop-mac
- Docker for Win - https://download.docker.com/win/static/stable/x86_64/
First clone the repo
git clone git@github.com:uktrade/dit-helpdesk.git
then using a terminal move inside of the folder:
cd dit-helpdesk
Copy the two development environment variables files:
cp .env.template .env
cp .env.template .env.test
and add entries where necessary (see comments for guidance). It may be easier to create one of the files and fill in the values, then copy that to create the second file.
You will need to access Helpdesk Vault to get the required environment variable secrets to use them in the file.
To do so you will need to generate a github personal access token. This is needed to log into vault.
Go here: Vault click Generate new token
and make sure it has these scopes: read:org
, read:user
.
Once you've done that, head over to Vault and login with the token. You'll need to select github
as your login option.
The values
HELPDESK_GA_GTM
HELPDESK_GA_UA
DEFRA_EMAIL
DEFRA_CONTACT
BEIS_EMAIL
BEIS_CONTACT
are not present in Vault, but the local environment will work without them.
The value for POSTGRES_PASSWORD
is also not present in Vault. You can provide your own value for this.
Install the pre-commit hook which will execute Python and JS code formatters before commit.
(If you do not already have the pre-commit
command installed, follow the directions at https://pre-commit.com/)
pre-commit install
Make sure that Docker is installed and running.
docker-compose up -d
This initial setup will take about an hour (depending upon machine and internet speed) to set up and fully import all content. On subsequent runs it will on take a minute or so to be up and running for development.
Before starting setup, download the required Rules of Origin files by following the instructions given below
in the section on the import_rules_of_origin
management command.
Run the following commands to activate a shell into the docker instance for the trade helpdesk app with the command.
docker-compose exec helpdesk /bin/bash
pipenv shell
./manage.py migrate
./reload_data.sh
The site will be available at http://localhost:8000/choose-country/
There may be a short delay before the site is available as the helpdesk
container waits for the
ElaticSearch container es
to complete initialisation. The command
docker-compose logs helpdesk | grep 'Development server is running'
will return success when the web server has started (until the logs are rotated).
You may see messages mentioning NewConnectionError
while the reload_data
script runs. These can be ignored.
When a management command (including reload_data
) completes successfully, you may see an error message that
Closing the transport connection timed out
. This can be ignored.
The setup process may fail in the prepare_import_data
step with the error "Found missing descriptions in [UK or EU]".
Steps for resolving this are described below in
the section on the prepare_import_data
management command.
You will need to have the following installed:
- Postgres
- Elasticsearch
- Redis
First we need to install GOV.UK Frontend and GOV.UK country and territory autocomplete (Which will also also install the required Accessible Autocomplete dependency), and other front end dependencies.
This is all done by going to the project root folder, which contains package.json
. Then run:
npm install
This will install all of the packages needed to build the front end static assets.
To build and move all of the static assets:
npm run build
You will need to install pipenv
pip install pipenv
Then install the required dependencies
pipenv install --dev
Then the init script can be run via
pipenv shell
./reload_data.sh
clone the directory
git clone https://github.com/uktrade/directory-forms-api.git .
Follow installation and setup instructions in https://github.com/uktrade/directory-forms-api/blob/develop/README.md to get the directory forms API running locally and to create a superuser.
Access the application admin screens locally in you browser with the url http://localhost:8011/admin.
Click the add button in the Client section and add helpdesk
as the name of the client then click submit. This will
generate the user identifier and access key that you need to add to the .env file DIRECTORY_FORMS_API_*
variables for the dit_helpdesk
application.
To run the tests in the Docker environment shell into the running container:
docker-compose exec helpdesk /bin/bash
pipenv shell
From within the docker shell terminal run the following command for full tests:
./manage.py test dit_helpdesk --settings=config.settings.test
for testing a single app run e.g. the hierarchy app:
./manage.py test hierarchy.tests --settings=config.settings.test
for testing a single app's test module run e.g. the test_views in the the hierarchy app:
./manage.py test hierarchy.tests.test_views --settings=config.settings.test
and so on.
for coverage reports run any of the above commands via coverage e.g.
coverage run manage.py test dit_helpdesk --settings=config.settings.test
and then the reports can be generated:
coverage html
will write the reports to test-reports/coverage_html
(configured in .coveragerc
).
You will then be able to access the coverage report html from within your project folder's root
from your host machine in test-reports/coverage_html
.
To write the reports to another folder, use:
coverage -d path/to/another_folder html
You will need to install pipenv stated above. Ensure Docker is running.
Update the version number in the Pipfile. There are 2 module lists; dev-packages
and packages
. The dev-packages
are ONLY loaded in local
docker environments, while changes to the packages
list are not always reflected in docker builds. Remember to test any updates locally and
on the dev environment by deploying the branch.
Get into the docker container by running the following:
docker-compose exec helpdesk /bin/bash
Generate updates to the Pipfile.lock file by running the following in the container:
pipenv lock
Do not manually update the Pipfile.lock file! Always auto generate it for the correct hashes!
./manage.py pull_api_update
This pulls the initial data from the trade tariff service API and stores the data in json files for future processing.
This will download the data for both the EU and the UK.
./manage.py prepare_import_data
This takes the information downloaded from pull_api_update
and transforms it by building the correct parent/child relationships in the data and removes any duplication.
This information is stored in json files.
This step may fail with the error Commodity found with no description: [commodity code]
due to a missing description
in the data downloaded from the Trade Tariff API by pull_api_update
.
The team responsible for the Trade Tariff API should be notified of this so they can fix their data.
In the interim, the commodity code given can be used to find the correct description from public data sources.
This can then be added temporarily to the dict dit_helpdesk.trade_tariff_service.HierarchyBuilder.PATCHED_DESCRIPTIONS
with the commodity code as the key and the description as the value. That will allow prepare_import_data
to run to completion (or to the point at which another description is found to be missing).
./manage.py prepare_search_data
This prepares the data from the prepare_import_data
step by converting the json data into a csv format.
./manage.py scrape_section_hierarchy
This creates the database instances of the nomenclature tree based on the data gathered from prepare_import_data
.
This will generate the full nomenclature trees for the UK and the EU.
This doesn't activate the newly created tree but generates it ready for it to be activated in other steps.
./manage.py import_rules_of_origin
Imports the rules of origin into the database. After the rules of origin are fully imported the rule text is post-processed for display purposes e.g. to display hyperlinks to the commodities a rule pertains to.
The rules of origin files are stored in an S3 bucket and this command will download these files and produce rules of origin objects tied to the currently active nomenclature tree. For docker environments an instance of Minio is used in place of connecting to the S3 bucket. To set up the rules of origin in your local docker, you will need to follow these steps:
1. Download & install CyberDuck
2. Access Vault and from helpdesk/dev get the following secrets:
```
ROO_S3_ACCESS_KEY
ROO_S3_BUCKET_NAME
ROO_SECRET_ACCESS_KEY
```
3. Open CyberDuck and click the Plus symbol button on the main page to create a new connection
4. In the popup window, fill in the following details:
- Select “Amazon S3” from the first dropdown box
- Type the ROO_S3_ACCESS_KEY into the Access Key Id box
- Type the ROO_SECRET_ACCESS_KEY into the Secret Access Key box
- Type the ROO_S3_BUCKET_NAME into the Path box
5. With the connection created, double click on the connection to open the S3 bucket
6. From this page copy the “rules_of_origin” folder to your local machine
7. Move the rules_of_origin folder to the following path in the dit-helpdesk repo:
```
volumes/s3/import-rules-of-origin-bucket
```
8. Access Vault and from helpdesk/docker_development get the following secrets:
```
S3_URL
ROO_S3_ACCESS_KEY
ROO_S3_BUCKET_NAME
ROO_SECRET_ACCESS_KEY
```
9. Copy these secrets into your local .env file and restart the container
10. Open the dit-helpdesk-helpdesk_1 container command line and shell in
```bash
pipenv shell
```
11. Run the command:
```bash
./manage.py import_rules_of_origin
```
./manage.py migrate_regulations
This will link regulations that are attached to the previous nomenclature tree to the new tree which is to be activated (which will have been generated in other commands).
./manage.py export_scenarios -o YOUR_FILENAME_HERE.csv
Will create a csv file containing the details of all the countries trade agreement status currently in the DB. Contains the following information: "Country code" - 2 digit code used to represent country "Country name" - Name of the country "UK agreement status" - indicator to show if the country in question has a trade agreement with the UK "EU agreement status" - indicator to show if the country in question has a trade agreement with the European Union "Scenario" - Code indicating what category of trade relationship this country has with the UK "GOVUK FTA URL" - URL address of the trade agreement details page on the .gov website "Trade Agreement Title" - The name of the trade agreement between this country and the UK (if applicable) "Trade Agreement Type" - The type of trade agreement between this country and the UK (if applicable)
./manage.py generate_search_keywords -f search/data
This generates the search keywords for the nomenclature tree.
Outputs the keywords into a csv file.
./manage.py import_search_keywords -f output/keywords_and_synonyms_merged.csv
Imports the search data from generate_search_keywords
into the database models for the nomenclature tree.
./manage.py swap_rebuild_index
Creates a new search index for the most recent nomenclature tree and swaps it out at the end.