-
Notifications
You must be signed in to change notification settings - Fork 0
Sesssion Sept
Day starts with checking correspondences
uses projects on github to create project boards
merge conflicts - pull requests
when there is an issue so for
change history you should use a --force flag. so sometimes when you realsie that you didn't need a commit
the conflict has been fixed
the pull request has now been resolved
git stash
is a way to park changes so when you want to keep the change around but you dont want to get rid of it it keeps it in a temp area
to get it back run git stash pop
it's about packaging the site in a way that everyone has the same code and the same site. so for example it has all of the dependencies etc.
The docker image is the base template. When you run the image, that gives you the container!! you specify the image with a docker file. This has an series of instructions and it ends with an ENDPOINT which gives the command that they need to run to start the template. So each dev is given that command to 'activate'/ run their container
have a repo of dockerfiles, which are pushed to docker hub which is a central repo
python tests have dependencies there are base images ... build a docker image himself and this will automatically be pushed to dockerhub, (python tests in docker) docker is really good for trying new things
source data then the transformer takes that in the elastic search and then API (pipeline) is a series of application and each one writes to a queue and then goes to the next DL queue stands for dead letter queue this means that we have the queue and the app will process messages but if then messages cant be processed it will be passed to the DL queue instead
AWS has a list of all the queues and includes how many messages on each queue
cloud watch - amazon app used for login, need to look at the transformer logs
an error came from the code his was working on yesterday. as soon as you detect and error, propograte immmediately, thore it up and allow it to be someone elses problem - zen of pythom
its always best to error as soon as possible,
the messages contain location, version etc. the location block tell us where that think is. he queries it
the error is that theres no tag so therefore it can't build a label.
the message is in a JSON file and the key is ... and had the json string
there are one message passed through the pipeline but each stage will process it slightly differently, so new json documents will be processed everytime.
slack integration
metrics on AWS measure all diff aspects of the pipeline It's also possible to write your own metrics - some metrics are purely informational and some - things are useful to tract
cloud watch can track the value of metrics and it has alarms. so you can threshold and say if this metric hits a particular point then thats bad.
one metric tracts the DL queue, if the number of messages is great than 0, ring the alarm. then it goes red and is in alarm state
when an alarm triggers, then it set to share the notification (stream) - for stuff like are the queue messages failing etc
LAMBDA
cloud server - you aren't responsible for provisioning the hardware - which has a slack API endpoint. so it goes through a series of tasks and then sends the message to slack.
collections have been on AWS for about 2 years but
slack channel with information about the state of the app
s3 is simple storage service and its a place to store large files, it comes with a console which is divided into buckets which comes with files
so the repo is for the code, but the s3 contains images, static collection images, private work flow, login details and his highly available and resilient - cross region replication and it can be protected against failure or loss digitally
index on elastic search is a collection of documents
the API allows us to choose which index we want to look at v1 api is now frozen so the indexes have worked so they can switch to the new API
for the api, there is a stage and a prod api the staging api runs in same place as the catalogue api it allows them to test api on stage, make sure that it's running as it should the they can switch configuration, and then it will run on production and it also means that is there a ch
romulus is that staging api remus is the production api
told the staging api that
terraform is used to define AWS functions, there are two steps, planning telling what to do and apply stage
so it's deployed after using terraform
ECS
health checks are run when a new task definition is created. If the new one run correctly then the old ones are deleted
in progress
the cataloguing is done, the indexing is fully populated managed mostly by devs with pm
src > pipeline > elastic search > API
digitisation at wellcome: in house photographers who photograph items and we need to be able to process
GOOBI digitisation workflow manager. a photographer uploads the images to goodbi and the rhiana adds metadata and then the goobi spits aout goobi and metadata. this needs to be archived
the archival service, so basically the content goes in and out comes and S3 bucket of assets and a DB of metadata. someone whos looking at the site can quierey the databases
the way it works goobi creates a bagit , and then posts doobi uploads everything to a S3 bucket and makes a post/ingest to an API controlled by the team
post > ingest > archivist > makes two back ups of the bag> so we know they are safely protected > unzip and verify ( this contains a manifest a list of everything thats in the archive, so you can verify that the archive is correct to use it) > registrar ( makes sure that the bagit file is correct and is backed up) > storage manifest (our own description of the files that we'd like to create, so we know what we've got) this is called a micro services application
transform in the json transforms it into a different blob of json DDS DLCS digerati built this
Polyglot development