NOTES.org

Things to do

Eliminate need to “navigate” in order for taggings to be sent to backend

This is a pitfall for users.

Old things to do

• cleanup FieldSel
• scattergl wtf
• Add #BC plot
• Add custom comments
• Add water pools
• Add purple points
• Fill missing entries in dq_db; re-check to 68000
• Consider tagging runs with blocked triggers
• Remove -1 when calculating plotAverage

Deployment checklist [9/9]

• [X] Build/upload frontend
• [X] Update Pipfile.lock
• [X] Dump dq_db
• [X] Delete/restart dybvdq-mysql
• [X] Load dq_db (convert to InnoDB?)
• [X] Build indexes/derived tables
• [X] Clean up app_db if necessary
• [X] Set env vars
• [X] docker-compose rm; docker-compose up –build

New deployment checklist

• [ ] git pull
• [ ] docker-compose up -d [–build –force-recreate] [nginx]
• [ ] scripts/db_update.sh if necessary

Frontend update

ssh root@dybdq.work “rm -rf visual_dq/dybvdq/front/build” && scp -r build root@dybdq.work:visual_dq/dybvdq/front && ssh root@dybdq.work “docker restart dybvdq-nginx”

Collaborative debugging

upnpc -a 192.168.1.2 3000 3000 TCP Add “0.0.0.0 lunix.mooo.com” to /etc/hosts HOST=lunix.mooo.com npm start Invite assistant to lunix.mooo.com:3000 Also look into disableHostCheck option to avoid using $HOST?

Docker workflow

On aftershock

git pull

docker build -t dayabay/dybvdq . docker run -d –net=host -v .:/app dayabay/dybvdq iptables -A IN_public_allow -p tcp -m tcp –dport 5000 -m conntrack –ctstate NEW -j ACCEPT

On DigitalOcean (old)

cd visual_dq git clone mkramer@edison.nersc.gov:bak/dybvdq.git

scp -r physics/dqwg/new_website/dybvdq/dybvdq-front/build root@142.93.95.86:visual_dq/dybvdq/dybvdq-front

cd visual_dq/dybvdq git pull docker container rm dybvdq-app docker image rm dayabay/dybvdq-app docker build -t dayabay/dybvdq-app . docker run –name=dybvdq-app -d –net=host -v $PWD:/app dayabay/dybvdq-app

On DigitalOcean (new)

cd ~/physics/dqwg/new_website/front npm run build

scp -r build root@dybdq.work:visual_dq/dybvdq/front

cd visual_dq/dybvdq git pull docker-compose stop docker-compose up –build

Inspecting a crashed container

(doesn’t seem to work) docker commit containername newimagename docker run -it –entrypoint /bin/bash newimagename

Database stuff

Making a snapshot

mysqldump -h dybdq.ihep.ac.cn -u dybrw -p --opt dq_db DqDetectorNew DqDetectorNewVld DqLiveTime DqLiveTimeVld most_recent_file_tag > dq_db.20180913.sql
mysqldump -h dybdb1.ihep.ac.cn -u dayabay -p --opt --skip-lock-tables offline_db DaqRawDataFileInfo DaqRawDataFileInfoVld > DRDFI.20181017.sql
# XXX confirm dybdb1 vs dybdb2, --skip-lock-tables, etc.

– takes < 1 minute on a good day Recommend using guwq@dybdq.ihep.ac.cn

Running on aftershock

Then scp the .sql to dayabay@aftershock.lbl.gov:visual_dq/mysql/dumps. It seems faster to transfer from dybdq than from lxslc6. What about mysqldump’ing directly onto aftershock? Bad idea; locks tables for too long.

docker run -d –name=vdq-mysql –user=1000:1000 –env=”MYSQL_ROOT_PASSWORD=***REMOVED***” -p 6603:3306 -v ~dayabay/visual_dq/mysql/conf.d:/etc/mysql/conf.d -v ~dayabay/visual_dq/mysql/data:/var/lib/mysql mysql:5.7

Running on DigitalOcean

docker stop dybvdq-mysql docker rm dybvdq-mysql rm -rf ~/visual_dq/mysql/data && mkdir -p ~/visual_dq/mysql/data

cd ~/visual_dq/mysql docker-compose up -d docker cp dumps/dq_db.20180904.sql dumps/DRDFI.20181017.sql dybvdq-mysql:/ docker exec -it dybvdq-mysql /bin/bash

mysql -p dq_db < dq_db.20180904.sql mysql -p dq_db < DRDFI.20181017.sql rm /*.sql

The load took around 8 minutes on DigitalOcean, 3 minutes on dellite! Remember to recreate the indexes! XXX convert any tables to InnoDB?

Indexes and derived tables

See extra/indexes.sql.

Can delete the following:

create index drdfi_runno_fileno on DaqRawDataFileInfo (runno, fileno); create index idx_runfile on DqDetectorNew (runno, fileno);

create table runno_fileno_sitemask (runno integer, fileno integer, sitemask integer not null, streamtype varchar(32) not null, primary key (runno, fileno));

insert into runno_fileno_sitemask select distinct runno, fileno, sitemask, streamtype from DqDetectorNew natural join DqDetectorNewVld left join DaqRawDataFileInfo using (runno, fileno) where streamtype != ‘AllStreams’ and sitemask=1 and runno between 21221 and 67976-1 and detectorid <= 2;

insert into runno_fileno_sitemask select distinct runno, fileno, sitemask, streamtype from DqDetectorNew natural join DqDetectorNewVld left join DaqRawDataFileInfo using (runno, fileno) where streamtype != ‘AllStreams’ and sitemask=1 and runno > 67976 and detectorid=2;

insert into runno_fileno_sitemask select distinct runno, fileno, sitemask, streamtype from DqDetectorNew natural join DqDetectorNewVld left join DaqRawDataFileInfo using (runno, fileno) where streamtype != ‘AllStreams’ and sitemask=2 and runno between 21221 and 26694-1 and detectorid=1;

insert into runno_fileno_sitemask select distinct runno, fileno, sitemask, streamtype from DqDetectorNew natural join DqDetectorNewVld left join DaqRawDataFileInfo using (runno, fileno) where streamtype != ‘AllStreams’ and sitemask=2 and runno > 26694 and detectorid <= 2;

insert into runno_fileno_sitemask select distinct runno, fileno, sitemask, streamtype from DqDetectorNew natural join DqDetectorNewVld left join DaqRawDataFileInfo using (runno, fileno) where streamtype != ‘AllStreams’ and sitemask=4 and runno between 21221 and 26694-1 and detectorid <= 3;

insert into runno_fileno_sitemask select distinct runno, fileno, sitemask, streamtype from DqDetectorNew natural join DqDetectorNewVld left join DaqRawDataFileInfo using (runno, fileno) where streamtype != ‘AllStreams’ and sitemask=4 and runno > 26694 and detectorid <= 4;

create index idx4lookup on runno_fileno_sitemask (sitemask, streamtype, runno, fileno);

MyISAM -> InnoDB

• https://support.severalnines.com/hc/en-us/articles/212426983-Migrate-MyISAM-tables-to-INNODB-using-mysqldump
• alter table DqDetectorNew engine=innodb;

Replication junk

(OLD) mysql> show master status; -----------------------------+--------------+------------------+

File

Position

Binlog_Do_DB

Binlog_Ignore_DB

-----------------------------+--------------+------------------+

mysql-bin.000263

494427353

-----------------------------+--------------+------------------+ 1 row in set (0.00 sec)

----------------------------+--------------+------------------+

File

Position

Binlog_Do_DB

Binlog_Ignore_DB

----------------------------+--------------+------------------+

mysql-bin.000266

637

----------------------------+--------------+------------------+ 1 row in set (0.00 sec)

https://www.digitalocean.com/community/tutorials/how-to-set-up-master-slave-replication-in-mysql

Old credentials from app.py

# DB = MySQLdb.connect(host='aftershock.lbl.gov', port=6603,
#                      user='root', passwd='***REMOVED***', db='dq_db')
# DB = MySQLdb.connect(host='dybdq.ihep.ac.cn',
#                      user='dayabay', passwd='***REMOVED***', db='dq_db')

HIGH AVAILABILITY

• google: nginx pause all requests
• https://github.com/solso/insomnia
• https://github.com/basecamp/intermission
• https://stackoverflow.com/questions/47366214/how-do-i-add-the-lua-module-for-nginx-on-alpine-linux (google: docker nginx lua)
• https://serverfault.com/questions/654780/how-to-suspend-nginx-requests-during-backend-upgrades
• https://serverfault.com/questions/675166/nginx-reverse-proxying-no-downtime-backend-update
• https://forum.nginx.org/read.php?2,177,177#msg-177

Let’s Encrypt

(google certbot wildcard) https://blog.miguelgrinberg.com/post/running-your-flask-application-over-https https://levelup.gitconnected.com/how-to-get-certbot-wildcard-certificates-3d25618a81e0

Initial setup:

git clone https://github.com/certbot/certbot cd certbot ./certbot-auto –os-packages-only ./tools/venv.sh # only needed once; creates venv source venv/bin/activate

Creation/renewal (no need to source virtualenv):

./certbot-auto -d dybdq.work -d *.dybdq.work –manual –preferred-challenges dns-01 –server https://acme-v02.api.letsencrypt.org/directory certonly cd ~/visual_dq/dybvdq/deploy docker-compose up -d –force-recreate –no-deps nginx # add –build it if doesn’t work

Renewal (every 90 days; due 2019/3/8): Either,

• Manually repeat the above
• Write a –manual-auth-hook to repeat the above
• Switch to the “standalone” certbot plugin (no more wildcard)
• Use the certbot-dns-digitalocean plugin

Certs/keys in /etc/letsencrypt

Password auth

https://www.digitalocean.com/community/tutorials/how-to-set-up-password-authentication-with-nginx-on-ubuntu-14-04

Nginx stuff

• https://stackoverflow.com/questions/46880853/deploy-create-react-app-on-nginx

SQLAlchemy

Executing on a specific bind:

• pallets-eco/flask-sqlalchemy#107

Compiling an on_duplicate_key_update statement

stmt = mysql.insert(Tagging).values(update) \ .on_duplicate_key_update(hall=Tagging.hall) text = stmt.compile(mysql.dialect()).__str__() db.get_engine(bind=’app_db’).execute(stmt or text)

Query example

D = DqDetectorNew D.query.with_entities(D.PLIKECOUNTS).filter(D.RUNNO == 54606 and D.DETECTORID == 1).all()

References

• https://github.com/Microsoft/TypeScript-React-Starter
• https://github.com/sw-yx/react-typescript-cheatsheet
• https://levelup.gitconnected.com/ultimate-react-component-patterns-with-typescript-2-8-82990c516935

Opening the project in VS Code

So far succeeded with opening dybvdq-front folder. Haven’t tried opening the whole chooch.

Updating type definitions

Just run typesync

Linting setup

Before:

"extends": ["tslint:recommended", "tslint-react", "tslint-config-prettier"],

After:

"extends": [],
"defaultSeverity": "warning",

“after”: [“y”, “y”, “P”, “k”, “>”, “>”, “C”]

From RunAndFile.tsx (ViewProps)

// Event handlers must return any instead of void
// https://github.com/piotrwitek/react-redux-typescript-guide#caveat-with-bindactioncreators
// onChangeRunno: (e: React.ChangeEvent<HTMLInputElement>) => void;
// onChangeFileno: (e: React.ChangeEvent<HTMLInputElement>) => any;
// onClick: (e: React.MouseEvent<any>) => any;
onChangeRunno: React.ChangeEventHandler<HTMLInputElement>;
onChangeFileno: React.ChangeEventHandler<HTMLInputElement>;
onClick: React.MouseEventHandler<any>;

Contents of src/components/App.test.tsx

import * as React from 'react';
import * as ReactDOM from 'react-dom';
import App from './App';

it('renders without crashing', () => {
  const div = document.createElement('div');
  ReactDOM.render(<App />, div);
  ReactDOM.unmountComponentAtNode(div);
});

Contents of src/App.css

.App {
  text-align: center;
}

.App-logo {
  animation: App-logo-spin infinite 20s linear;
  height: 80px;
}

.App-header {
  background-color: #222;
  height: 150px;
  padding: 20px;
  color: white;
}

.App-title {
  font-size: 1.5em;
}

.App-intro {
  font-size: large;
}

@keyframes App-logo-spin {
  from { transform: rotate(0deg); }
  to { transform: rotate(360deg); }
}

Contents of src/index.css

body {
  margin: 0;
  padding: 0;
  font-family: sans-serif;
}

JS snippets

toQuerystring

export const toQuerystring = (obj: object) =>
  Object.entries(obj)
    .map(([k, v]) => `${k}=${encodeURIComponent(v)}`)
    .join('&');

TypeScript References

Variadic generics

microsoft/TypeScript#5453

Removing ‘undefined’

microsoft/TypeScript#24067

Strict bind, call, apply

microsoft/TypeScript#27028

Alternative backend frameworks

Kotlin

• http4k
• vert.x
• ktor
• jooby
• javalin

Misc notes

SSH hopping

• https://serverfault.com/questions/337274/ssh-from-a-through-b-to-c-using-private-key-on-b (google: ssh use key from jump)

Plotly

autosize, width etc.

• plotly/react-plotly.js#76

Versioning

• https://www.npmjs.com/package/genversion
• https://coderwall.com/p/mk18zq/automatic-git-version-tagging-for-npm-modules
• https://stackoverflow.com/questions/9153571/is-there-a-way-to-get-version-from-package-json-in-nodejs-code
• Google: npm include git tag
• Google: npm get version from package.json

Potentially useful libraries

• immer
• reselect
• redux-starter-kit
• bulma (CSS)

Long-term refactors

• Replace Redux with Context
• Use React Hooks instead of class components