Diabetes portal for the Medical Population Genetics department at the Broad Institute
Switch branches/tags
${BRANCH} Final-SIGMA-fixes GoogleAuthentication IBD-theming Maria-adding-new-data-section Maria-editing-contact-page Maria-editing-data-page Maria-editing-project-pages Maria-updating-contact-page Maria-updating-data-page MiVersion QA-28-refs/tags/DEV_OK-113-087ba QA-29-refs/tags/DEV_OK-122-a39e8 QA-30-refs/tags/DEV_OK-124-f7fff QA-31-refs/tags/DEV_OK-131-0b67e QA-32-refs/tags/DEV_OK-132-a1e0a QA-33-refs/tags/DEV_OK-133-16a98 QA-34-refs/tags/DEV_OK-184-c022a QA-44-refs/tags/DEV_OK-113-087ba QA-45-refs/tags/DEV_OK-113-087ba QA-47-refs/tags/DEV_OK-113-087ba Spanish_unicode_fix StrokeRCV1.0 TraitInfoOldApiIndependent abcde als_portal amp_redesign arz_chooser_dynamic_columns ashgDemo assoc-at-glance biAllelicCapable branch_150623_2a93f_35 branch_150627_fef49_36 branch_150701_84b63_37 branch_150705_cc831_38 branch_150705_2c800_39 branch_150706_cf39e_40 branch_150707_3bbb5_41 branch_150708_d74bf_42 branch_150709_02b6f_43 branch_151020_cddf5_45 branch_151104_57672_46 branch_151201_71692_48 branch_151202_87c04_49 branch_151207_cfa7e_53 branch_151207_cfa7e_54 branch_151207_cfa7e_55 branch_151208_b5fe0_65 branch_151208_cfa7e_64 branch_151211_ffd8c_66 branch_160123_13d2b_67 branch_160125_ae9a4_68 branch_160127_c730b_69 branch_160127_f2ffe_70 branch_160201_cba73_71 branch_160212_a23c9_72 branch_160314_f9094_73 branch_160317_8d896_74 branch_160318_0239c_75 branch_160328_24fd7_76 branch_160331_be6c1_77 branch_160331_98cd7_78 branch_160331_8812e_79 branch_160401_c4f8f_80 branch_160401_172ca_81 branch_160411_98071_82 branch_160413_5cd31_83 branch_160425_f2092_85 branch_160427_60487_86 branch_160429_f943e_87 branch_160429_7b8a4_88 branch_160505_28fa2_89 branch_160506_93075_90 branch_160513_11da9_91 branch_160516_6b813_92 branch_160517_7b383_93 branch_160608_43e26_94 branch_160608_79feb_95 branch_160609_a9688_96 branch_160609_61eda_97 branch_160616_50fe3_99 branch_160616_6557a_98 branch_160620_acf3d_100 branch_160628_c786f_101 branch_160726_4416a_102 branch_160818_e76dd_103 branch_160901_28140_104 branch_160912_d3d01_105 branch_160923_ae10b_107 branch_161003_fe703_108 branch_161003_32c7c_109 branch_161004_b53ca_110 branch_161005_f251e_111 branch_161007_710ee_112 branch_161115_e560e_113 branch_161116_66ded_114 branch_161121_1f5ce_115 branch_161122_2d63c_116 branch_161202_0aac8_117 branch_161212_e0c59_121 branch_170110_5f64d_122 branch_170117_75bf8_123 branch_170214_02b3d_124 branch_170310_22c20_125 branch_170329_842eb_127 branch_170330_68428_128 branch_170404_d07bc_130 branch_170404_d5288_129 branch_170405_b93d3_131 branch_170405_88fbd_132 branch_170421_a4995_133 branch_170424_a8f3d_134 branch_170429_df3bf_137 branch_170501_a948e_138 branch_170501_58b52_139 branch_170502_400b8_141 branch_170503_702a2_142 branch_170504_aa1a8_144 branch_170519_2ee46_145 branch_170519_93028_146 branch_170524_fc215_150 branch_170525_8467c_151 branch_170530_616b2_152 branch_170601_91ec7_153 branch_170605_6ec1d_155 branch_170605_0849b_154 branch_170606_db192_156 branch_170607_80043_157 branch_170608_bb3b1_158 branch_170608_6c002_159 branch_170609_fa602_161 branch_170609_1ee89_160 branch_170609_49bb3_162 branch_170616_8b765_163 branch_170801_ccbe8_164 branch_170802_696cd_165 branch_170814_c1323_166 branch_170831_c1323_167 branch_171109_e88de_168 branch_171111_dfd30_169 branch_171201_ca855_170 branch_171211_86e9e_171 branch_171212_1d5c7_172 branch_171221_79ca5_174 branch_171221_562f6_173 branch_180108_acd53_175 branch_180108_cc11a_176 branch_180112_c57c9_179 branch_180112_e3f34_177 branch_180112_e3f34_178 branch_180122_8d4cd_181 branch_180125_20c61_182 branch_180209_04e7f_183 branch_180209_04e7f_184 branch_180220_4805e_185 branch_180221_eb130_186 branch_180226_387b4_187 branch_180227_b5fdc_189 branch_180227_82acd_188 branch_180227_58517_190 branch_180302_9a901_191 branch_180312_f8a0f_192 branch_180321_cda67_194 branch_180321_fc4a4_193 branch_180327_a8c9e_195 caseControlDistinctions chooser_human_friendly_nammes chooser clumping_fix config_groovy configBasedPortals configBasedPortals2 credSetPointColoring data-page demo52k dk-dynamicUiBase dk-findmore-ui dk-findmore-ui2 dk-grs-page-formatting dk-modules-page dk-modules-rolledback-safe dk-modules-rolledback dk-onthology-search dk-ontology-search-phePlot dk-phenotype-search dk-phenotype-search2 dk-sleep-on-front dk-static-content dk-traits-table-ui dk-variant-finder dk_gait_tabs_ui dk_mockup_cleanup dynamicFilters dynamicTissuePlots dynamicTissueWithCredSetHandling dynamicUiBase dynamicVariantResultsTable epiGraph epiGraph2 ethTable explicitlySpecLZSet exptWithLZ external_auth feature/rangeQueryAggregated fix-beta-oddsratio forestPlot forestPlotScramble functionalData gaitAllTab gaitBurden gaitUIStudy geneBasedComp geneInfoWithAjax genePageFixes genePageForRelease googleAuthExperiments googleAuthentication grsModule gwas_summary gwasSummaryRevised hailMetaData hiGlassExptWorksForV2f higlassExpt html_popovers i18n_translations ibdRefine ibdTestVer ibdVer2 igvExpt2 igvExpt3 igvSuccessAtLast js-exp jstreeGridExpt ky_beacon_addressing_feedback ky_beacon_deployment ky_beacon_json_text_toggling ky_beacon_testing_and_user_input_validation legalese limitOneQuery linefillplot loginTracking lz-dropdown lz-finalization lz-fixes lz_update_0.7.0 lzDiffDatasets lzExpts2 manhattanChanges manyMinorNewGenePageChanges maria mariacos-patch-1 mariacos-patch-2 mariacos-patch-3 mariacos-patch-4 mariacos-patch-5 mariacos-patch-6 mariacos-patch-7 master masterMergeWithDynamicMdv masterPlusHier md_add_smart_api md_demo_intel md_ebi_amp_demo md_ebi_burden md_ebi_configuration md_ebi_portal md_fed_hoorn_dataset md_fed_var_id_fix_common md_grails_cache md_intel_rp md_stroke_portal201610 md_test_federated mergeTroublesDynamicVariantResultsTable mergingNewGenePage metaAnal metadata-translation modifiedUi more-projects new-home-page newApi newBurden newCredSetFunc newEpiPortalChanges newGeneInfo newIgvExpt newLZTry newprops no-pheno-opt noBurdenTestForMultiallelics oliver phewasForest phewasOption phewasTest portalIndep preeti-variant-finder prodIter2_aug21_2015 remo_samples remo_samples_2 removingOldApi saveSession savefromcrash search-builder-updates sep11_2015_prod sigma sigmaInMexicoDemo sleep-portal spanish-pagination spanish-properties-160126 spanish-translation-1 spanish-translations-for-sigma startTowardsNewUI startingDownPathToUKBB stroke_data_pages supposedFix systemizingEpigeneticMatchTypes tBurden tabbed-builder testAti testing-ignore traitOnNewApi traitsTable traitsTable2 translatebigwig ucsc_hack2 ucsd_hack ucsd_hack3 ui-ada unneededFromJun30 useAggregationTable v-a-table v2fGeneralizedExpt v2fgeneralizations v2fgeneralizations2 v2fgeneralizations3 variant-results-table variant-summary-polishing variant-table-fix variantSelection vrt-update ws_load_testing
Nothing to show
Clone or download
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
classes/production/dport-grailsPlugins saved files from crash Nov 13, 2017
data Incorporate the extra variant info. Required an additional Ajax call … Dec 5, 2014
grails-app updated number of datasets and phenotypes Dec 13, 2018
notes adding fed test portal log of deployments Jun 1, 2018
scripts add versioning information, along with a few scripts necessary to gen… Dec 30, 2014
src fix bug with data sets that combine credible set and P values. Also c… Oct 24, 2018
test fix broken test Oct 29, 2018
web-app adding DNCRI data page Dec 13, 2018
wrapper basic framework for a grails 2.4.2 project Jul 14, 2014
.classpath basic framework for a grails 2.4.2 project Jul 14, 2014
.gitignore work in progress Feb 16, 2016
.project basic framework for a grails 2.4.2 project Jul 14, 2014
Diabetes_i18n_translations_needed.xlsx first pass at completing missing i18n translations Nov 23, 2015
README.md version works for v2f, but has trouble on t2d? Sep 17, 2018
application.properties fix all but one of the bugs that Maria found prior to merge. The last… Oct 10, 2018
gpw merge, plus remove reference to dynamic burden test Jan 12, 2015
grailsw basic framework for a grails 2.4.2 project Jul 14, 2014
grailsw.bat basic framework for a grails 2.4.2 project Jul 14, 2014
hs_err_pid19532.log comment out unused tests for now May 19, 2017
users.tsv read in user list. Allow password reset with no username validation. … Aug 26, 2014

README.md

__Important note to portal developers and sysops __ __ The code requires the incorporation of a private configuration file, as described below in the section titled "Configuration". The code may compile correctly without private configuration files, but it will not run in any of the following configurations:

  • debug mode for development
  • test mode, for running unit, integration, or functional tests
  • generating a war file for the purposes of deployment
  • running that war file on your target server More information is provided down below (see the "configuration" section in this README file)__

Diabetes portal

The diabetes portal is written in Grails. The following description should give you all the information you need to download, compile, and run a local version of the portal.

Table of Contents:

Environment
Set up your environment for running a Grails based project
Get the code
Download the code from GitHub
Testing
Run some pre-packaged tests to make sure your environment is in good shape
Run the portal
Start up the diabetes portal and run it locally

Environment

The following steps should give you a working development environment:

Install Java

Googling “download Java jdk” should take you where you need to go. The same command brought me to this URL from which I successfully downloaded Java.

Note that you will need the whole Java development toolkit (JDK), not simply the Java runtime environment (JRE). As far as versions, I have successfully compiled this program both under Java 6 and Java 7 (I haven't yet tried Java 8).

Once you’ve downloaded Java you should run the installer. Make sure that one way or another you end up with an environmental variable named “JAVA_HOME” which points at the base directory of your JDK. I usually perform this assignment inside my .bash_profile with a few lines that look like this:

JAVA_HOME=/cygdrive/c/java7
PATH=$JAVA_HOME/bin:$PATH
export JAVA_HOME

Other approaches to defining an environmental variable are fine too. Simply make sure that if you type:

echo $JAVA_HOME

that you see the directory you expect to see. The critical test of your success in this step, however, is typing

java -version

which should report the version of your Java runtime environment.

Install GVM

There’s a tool called GVM that can do most of the rest of the installation work for you. You can find that tool at the following URL http://gvmtool.net/. You start by installing GVM, which has only two dependencies, namely ‘curl’ and ‘unzip’. These are essential-- if you don't have them installed then you cannot proceed with GVM. To be sure you can certainly install grails without GVM if you like. Check the instructions here.

Provided that you want to proceed with GVM, use the following command line evocation:

 curl -s get.gvmtool.net | bash

Once this line runs successfully the rest of the install becomes pretty easy.

Install Grails

with GVM in place installing grails should consist of exactly one line:

gvm install grails 2.4.3

Once this command completes successfully you should be able to run the following command from the command line (Note the extra - that the analogous Java command does not require):

grails –-version

and you should see a response that indicates that grails is working and running the version number you requested

A quick note for Mac users: homebrew also provides an easy approach to perform the Grails installation. And while we're on the topic installing Grails by hand is also quite easy. You download the package and set a GRAILS_HOME environmental variable and you're all set.

Install Groovy

Not strictly necessary (since grails comes with its own version of groovy built-in) but you may as well having come this far

gvm install groovy

If your Groovy installation completed successfully then the following command will tell you that Groovy is running

groovy --version

To be clear, you can run grails without installing Groovy explicitly. If you do install Groovy explicitly, however, then you can begin writing programs in straight up Groovy as well, which has its own attractions. Groovy is a really fun language with lots of possibilities. Check out the documentation here

Get the code

Now it’s time to pull down the code from the repository. Provided that you have already installed a git client this can be accomplished with one line

git clone git@github.com:broadinstitute/dig-diabetes-portal.git

Git will create a new directory called dig-diabetes-portal. Change your current working directory and then you will be ready to start building the system.

While your cloning repos note that both Groovy and Grails have their own repo, and that you can download the source code for either or both languages. Each of these repos is instructive and worth going through, especially if you get stuck somewhere along the way. For the record, note that Groovy is written mostly in Java (though there is currently a project underway to rewrite Groovy in Groovy). Grails, by comparison, is written mostly in Groovy.

Testing

A good place to start might be with running the tests. You can run the unit tests with the following command:

grails test-app unit:

You can run integration tests with this command:

grails test-app integration:

Skipping the 'unit' or 'integration' specification will cause the entire test suite to run. Conversely adding a class name to the end of one of the earlier commands will allow you to run only a single test.

Run the portal

Once you have demonstrated that you can run both tests to completion you can be reasonably confident that your environment is in good shape and that you are ready to run the portal. The command looks like this:

grails run-app

this command should be all you need to start a running application from the command line. If you'd like to do some development, however, you will probably want to import the project into an IDE. I like IntelliJ, though there is also an active community of Grails developers who use eclipse. If you would like to prepare your project for an IDE then there is a grails command for that. To open up the project in IntelliJ, for example, the command is:

grails integrate-with --intellij

which will generate an "*.ipr" file along with everything else you need to open the fully functional project under IntelliJ

If you want to now deploy a the portal you would need to use grails to create a WAR file that you could then the handoff to a suitable servlet container such as Apache or Tomcat. The grails command necessary to create a war file by hand is:

grails war

In order to increase consistency a bash script exists to build war files. This script is named 'gpw', and it passes a few parameters in as the war is created making the resulting executable easier to track. To run this script enter the following command from a bash-aware command prompt:

./gpw nameForThisVersion

Note that this script performs a 'git rev-parse HEAD' in order to determine your current git repo version, so your command line interface needs to recognize 'git'.

Configuration

While the diabetes portal is open source at its core, there are a few keys ( mostly relevant to authentication and services we pay for) which need to remain secret. As well, you may choose to override some of the default values utilized by the portal during compilation. In both cases you'll need to utilize a private configuration file that is stored somewhere on a local disk. The portal is looking for such a private configuration file. To determine the directory that should hold your private config, try running the above 'gpw' command and watch the resulting console output to find a line that look something like this:

>>>>>>>>>>>Note to developers: config files may be placed in the directory  = /Users/ben/.grails/dig-diabetes-portal

This line will tell you where on disk to store your private configuration file. In that directory you may then create a file named 'dig-diabetes-portal-commons-config.groovy'. This file will now be read in by grails during the compilation phase, and any values listed in this file will override those listed in the portals default configuration file (named Config.groovy). Any values not explicitly overridden will retain their default values. You will know that you have created a personal configuration file in the right place and with the right name if you see lines similar to the following when you compile the portal using gpw.

\*\* !! config override is in effect !! \*\*
\!\!\!\!\! file:/Users/ben/.grails/dig-diabetes-portal/dig-diabetes-portal-commons-config.groovy !! **

Note that a personal configuration file will (very) soon become mandatory, and that this file will need to contain at a minimum the following line:

oauth.providers.google.api.secret = 'xxxxxxx'

where xxxxxxx represents our Google OAuth secret key. For developers working on the diabetes portal in the Medical Population Genomics department at the Broad Institute then please see me (ba) for our current key. Otherwise set up your own authorization secret key with Google and replace xxxxxxx with your number.

Special note: if you don't create the above configuration file, or else if you put it in the wrong place then two things will happen: first, during compilation the Grails code will give you a polite, comprehensible message indicating that you did not supply a configuration file. That message will Show up on your console, and should look something like this:

\*\* No config override  in effect \*\*

as well, Grails will error out without running, and will provide a long and barely comprehensible error message that starts out looking like this:

Error |
Fatal error running tests: Error creating bean with name 'grails.plugin.databasemigration.DbdocController': 
Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'instanceControllerTagLibraryApi': 
Injection of autowired dependencies failed; nested exception is org.springframework.beans.factory.BeanCreationException: Could not autowire method: 
public void org.codehaus.groovy.grails.plugins.web.api.ControllerTagLibraryApi.setTagLibraryLookup(org.codehaus.groovy.grails.web.pages.TagLibraryLookup); 
nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'gspTagLibraryLookup': 
Invocation of init method failed; 
nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'grails.plugin.springsecurity.oauth.SpringSecurityOAuthTagLib': 
Initialization of bean failed; nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'oauthService': 
Invocation of init method failed; nested exception is java.lang.IllegalStateException: 
Missing oauth secret or key (or both!) in configuration for google. (Use --stacktrace to see the full trace)
.Tests FAILED 
|

So if you see the above message, look to see if you have a configuration file, if it is in the right place, and if it contains the right values.

Development tools:

Code coverage

```bash grails test-app -coverage ``` and see ./target/test-reports/cobertura

Automated code Checker

```bash grails codenarc ``` and find your summary report in ./target/CodeNarcReport.html

CI system and version promotion

We have a process to move from code push to production deploy. It looks like this:

  • 1)every time a developer pushes code:
  • 1.1) the CI system pulls the latest code from Git on the master branch
  • 1.2) the system runs all the unit and integration tests. IF they pass then
  • 1.3) CI deploys, and gives that version a CI Git tag. Then…
  • 2) every morning at 2 AM
  • 2.1) the tag describing last successfully deployed CI version is pulled from Git
  • 2.2) the system runs all the unit and integration tests. IF they pass then
  • 2.3) DEV deploys, and gives that version a DEV Git tag.
  • 2.4) developers(and potentially testers) look at the deployed DEV system, and decide whether they like it. If they do then eventually…
  • 3) a developer goes to the Jenkins project named 'MANUAL-deployedToQA'
  • 3.1) the developer goes in with the name of a DEV tag in hand. The developer chooses that tag from the drop-down list, then presses the 'Build' button
  • 3.2) the system runs all the unit and integration tests. IF they pass then
  • 3.3) the system BRANCHES the code, and deploys that branch to QA
  • 3.4) developers look at the system themselves, but also request the attention of testers/internal-users to see if this system is acceptable. If it is then...
  • 4) a developer goes to the Jenkins project named 'MANUAL-deployedToProd' (hopefully during nonpeak hours)
  • 4.1) the developer goes in with the name of a QA branch in hand. The developer chooses that branch name from the drop-down list, then presses the 'Build' button
  • 4.2) the system runs all the unit and integration tests. IF they pass then
  • 4.3) the system BRANCHES the code again, and deploys that branch to PROD
  • 5) and, as necessary...
  • 5.1) someone finds a critical bug that can't wait for the next iteration. A developer checks out the branch currently running on prod and fixes the problem.
  • 5.2) the developer checks in the fix. Since it isn't on the master branch it doesn't enter the CI system, so test carefully!
  • 5.3) tag your version by hand.
  • 5.4) Then follow steps (4) above to deploy the revised code
  • 5.5) Don't forget to merge your version back into the master branch before the next iteration begins!
done. Code has been deployed to PROD.

Note: This entire process should be completed once per iteration (or more, if any problems are discovered after the deployment). This deployment should happen at least a couple of days before the user group, so that the group can help take a look at the newly minted production code.

Install Jenkins Server

eg make install JENKINS_HOME=/humgen/diabetes/portal/dig-jenkins/test_server/home/ creates log in $JENKINS_HOME/Jenkins_${JENKINS_PORT}.log