Merge pull request #1 from magda-io/publishing-to-ckan

Publishing to ckan

.github/CONTRIBUTING.md

@@ -0,0 +1,62 @@
+This is a work in progress and will change as we flesh out the process better. If there's any suggestions you'd like to make, please make them :).
+
+# Getting Started
+
+Before you do anything naturally you'll want to start by making sure you understand the [README.md](https://github.com/magda-io/magda/blob/master/README.md) first.
+
+If you're writing code then you'll want to run through [the building and running documentation](https://github.com/magda-io/magda/blob/master/doc/building-and-running.md) to get yourself set up too.
+
+# How to Contribute
+
+How you contribute will depend on your relationship to the project:
+
+-   Core team - i.e. you're contributing a significant amount of time per week, probably as part of your job at Data61 (although we're very keen to collaborate with other organisations!)
+-   External Contributor - i.e. you're using Magda and want to contribute features or bug fixes on a more casual basis - outside of our sprints.
+
+## Core Team Members
+
+We make use of [ZenHub](https://www.zenhub.com/) to add extra project-management functionality into Github, so you'll want to have that installed (it comes as a browser extension) also - it's all free for public repos.
+
+Magda uses Github issues as its base unit of work - each user story, bug, chore etc should have an issue attached to it. Issues (particularly user stories or other issues related to adding features) are often grouped into epics, which are special ZenHub-only issues that group them.
+
+Magda currently works in 2-week sprints, tracked as milestones in Github. At the start of each sprint, we run a short sprint-planning session in which we decide what we're going to work on in the next sprint and estimate on each issue. Because issues have estimates attached to them, wherever possible they should be divided up into simple, discrete tasks - i.e. try to avoid creating mega-issues with many steps that will take weeks to get through, instead prefer to create multiple issues, potentially grouping them into an epic or using Zenhub's ability to make one issue dependent on another to group them together.
+
+### Developer Workflow
+
+1. Issue is created / written up, and labels applied - particularly apply the "bug" or "good first issue" labels if relevant
+2. Issue is added to a sprint during a sprint-planning meeting, moves to the _Backlog_ column
+3. Developer starts the issue - makes sure it's assigned to them, drags it to the _In Progress_ column
+4. Developer works on the issue, submits it as a Pull Request and drags it to the _Pull Request_ column
+5. PR is reviewed by another core team member, comments are fixed, approved
+6. PR is merged **by the reviewer** when they're happy, and the branch it came from is deleted.
+7. Reviewer drags the issue into the _QA_ column if there's a UI part to it that needs to be QA'd by the designer, otherwise go to step 10.
+8. `master` is deployed to dev (at the moment this is manual)
+9. Issue is QA'd by the designer - if something needs changing a comment is left on the issue `@`ing the developer and it's dragged back to _In Progress_, - go to step 4. Otherwise...
+10. Issue is dragged to the _Done_ column by the person who approved the QA.
+11. At the end of the sprint, all issues in `Done` are closed.
+
+Unfortunately this process is all manual at the moment, so it's important that you keep on top of manually moving your issues along the columns as they progress. As the process becomes more stable we'll look at creating some kind of bot to make things a bit less manual.
+
+## External Contributors
+
+We don't expect people outside the core team to go through the same process as core team members. The workflow for external contributions goes something like this:
+
+1. Make sure there's an issue for what you're doing - if you're thinking of sending us a PR for something, make an issue for it first so we can discuss it with you!
+2. Make sure you make a comment in the issue letting us know that you've started work.
+3. Fork the repo and make some commits
+4. Make a PR on `magda-io/magda` - keeping to the standards below
+5. Respond to PR review comments
+6. PR gets merged into `master`
+7. Changes become part of the next release!
+
+# Standards
+
+-   Where practical, changes should be accompanied by unit tests. Unfortunately there's still a lot of untested code in the code base - if you're modifying untested code, try to at least write tests to test _what you've changed_.
+    -   The front-end is (currently) an exception to this as it's still very experimental
+-   Javascript/Typescript must be formatted with [Prettier](https://github.com/prettier/prettier), Scala should be formatted with [Scalariform](https://github.com/scala-ide/scalariform).
+-   All contributors should be listed in CONTRIBUTORS.md - in your first PR, please make sure you've added yourself!
+-   All changes should be recorded as a bullet point in CHANGES.md, as part of the PR that made those changes.
+
+# Release Cycle
+
+Currently the release cycle for Magda isn't very clever - every time there's a new release we increment the version number and push new images to DockerHub. Releases aren't planned to any real schedule, although they should happen at least fortnightly.

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,40 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: ""
+labels: bug
+assignees: ""
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To Reproduce**
+Steps to reproduce the behavior:
+
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+
+-   OS: [e.g. iOS]
+-   Browser [e.g. chrome, safari]
+-   Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+
+-   Device: [e.g. iPhone6]
+-   OS: [e.g. iOS8.1]
+-   Browser [e.g. stock browser, safari]
+-   Version [e.g. 22]
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/feature-request.md

@@ -0,0 +1,19 @@
+---
+name: Feature Request
+about: Suggest an idea for this project
+title: ""
+labels: feature request
+assignees: ""
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context or screenshots about the feature request here.

.github/ISSUE_TEMPLATE/refined-issue.md

@@ -0,0 +1,23 @@
+---
+name: Refined Issue
+about: A self-contained bit of work to be done, usually produced through refinement
+title: ""
+labels: refined-unreviewed
+assignees: ""
+---
+
+## Description
+
+Briefly describe what needs to be done, any background etc.
+
+## Acceptance Criteria
+
+-   A bullet-point list of things that must be true before a PR for this issue can be merged
+
+## Technical Notes
+
+Anything kind of suggestions for implementation or other bits of info that need to be known to implement something technically.
+
+## Out of Scope
+
+-   A bullet list of anything that's out of scope for this issue - particularly things that are related but going to be done in future issues

.github/ISSUE_TEMPLATE/unrefined-issue.md

@@ -0,0 +1,9 @@
+---
+name: Unrefined Issue
+about: An issue that needs to be described in low-detail and refined later
+title: ""
+labels: unrefined
+assignees: ""
+---
+
+## Description

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,11 @@
+### What this PR does
+
+Fixes #<insert issue number here if relevant>
+
+This is a description of what I've done in this PR, especially explaining choices made to make the reviewer's job easier. If I don't replace this paragraph with an actual description, my PR will probably be ignored.
+
+### Checklist
+
+-   [ ] There are unit tests to verify my changes are correct or unit tests aren't applicable
+-   [ ] I've updated CHANGES.md with what I changed.
+-   [ ] I've linked this PR to an issue in ZenHub (core dev team only)

.github/workflows/main.yml

@@ -0,0 +1,44 @@
+name: Main CI Workflow
+
+on: push
+
+env:
+  REPO_NAME: magda-minion-ckan-exporter
+
+jobs:
+  build-test-docker:
+    name: Build, Test & Push to Docker
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Use Node.js 10
+        uses: actions/setup-node@v1
+        with:
+          node-version: 10
+      - run: yarn install
+      - run: yarn build
+      - run: yarn test
+      - name: Setup Helm
+        uses: azure/setup-helm@v1
+        with:
+          version: v3.2.0
+      - run: yarn helm-lint
+      - name: Login to GitHub Package Repository
+        env:
+          GH_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
+        run: docker login docker.pkg.github.com -u magdabot -p ${GH_TOKEN}
+      - name: Build Docker Image & Push
+        run: yarn docker-build-prod --repository=docker.pkg.github.com/magda-io/${REPO_NAME} --name=${REPO_NAME} --version=${GITHUB_SHA}
+
+  build-test-node-12:
+    name: Build, Test Only with Node 12
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+      - name: Use Node.js 12
+        uses: actions/setup-node@v1
+        with:
+          node-version: 12
+      - run: yarn install
+      - run: yarn build
+      - run: yarn test

.github/workflows/release.yml

@@ -0,0 +1,67 @@
+name: CI Release Helm Chart
+
+on:
+  release:
+    types: [published]
+
+env:
+  REPO_NAME: magda-minion-ckan-exporter
+
+jobs:
+  release-helm-chart:
+    name: Release Helm Chart
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Use Node.js 10
+        uses: actions/setup-node@v1
+        with:
+          node-version: 10
+      - run: yarn install
+      - run: yarn build
+      - run: yarn test
+      - name: Setup Helm
+        uses: azure/setup-helm@v1
+        with:
+          version: v3.2.0
+      - run: yarn helm-lint
+      - name: Login to GitHub Package Repository
+        env:
+          GH_TOKEN: ${{ secrets.GH_ACCESS_TOKEN }}
+        run: docker login docker.pkg.github.com -u magdabot -p ${GH_TOKEN}
+
+      - name: Build Docker Image & Push to Github Registry
+        run: yarn docker-build-prod --repository=docker.pkg.github.com/magda-io/${REPO_NAME} --name=${REPO_NAME}
+
+      - name: Login to Docker Hub
+        env:
+          DH_TOKEN: ${{ secrets.DOCKER_HUB_PASSWORD }}
+        run: docker login -u magdabot -p ${DH_TOKEN}
+
+      - name: Re-tag & Push Docker Image to Docker Hub
+        run: yarn retag-and-push --fromPrefix=docker.pkg.github.com/magda-io/${REPO_NAME}/ --fromName=${REPO_NAME}
+
+      - name: Configure Git
+        run: |
+          git config user.name "$GITHUB_ACTOR"
+          git config user.email "$GITHUB_ACTOR@users.noreply.github.com"
+
+      - name: Setup AWS CLi Tools
+        uses: chrislennon/action-aws-cli@v1.1
+
+      - name: Release Helm Chart
+        env:
+          CR_TOKEN: "${{ secrets.GH_ACCESS_TOKEN }}"
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+        run: |
+          cd deploy
+          mkdir -p sync_dir
+          mkdir -p index_dir
+          if ! aws s3 cp s3://magda-charts/index.yaml index_dir/index.yaml; then echo "failed to copy current repo index" && exit 1; fi
+          helm package -d sync_dir ${REPO_NAME}
+          helm repo index --merge "index_dir/index.yaml" sync_dir
+          mv -f sync_dir/index.yaml index_dir/index.yaml
+          aws s3 sync sync_dir s3://magda-charts/ --acl public-read
+          aws s3 cp index_dir/index.yaml s3://magda-charts/index.yaml --acl public-read

.gitignore

@@ -102,3 +102,8 @@ dist
 
 # TernJS port file
 .tern-port
+
+# VSCode files
+.vscode
+
+ckanApiKeysMap.json

Dockerfile

@@ -0,0 +1,6 @@
+FROM node:10
+
+RUN mkdir -p /usr/src/app
+COPY . /usr/src/app
+WORKDIR /usr/src/app/component
+ENTRYPOINT [ "node", "/usr/src/app/component/dist/index.js" ]