September 21-22, 2016
- Hack/Doc Topics:
- CSV Importer
- Islandora-in-a-Box
- Ben Rosner (Barnard - 9/21-22)
- Martha Tenney (Barnard - 9/21-22)
- Gavin Morris (Common Media - 9/21-22)
- Noah Smith (Common Media - 9/21)
- Patrick Dunlavey (Common Media - 9/21-22)
- Mark McFate (Grinnell - 9/21-22)
- Aaron Neslin (Hamilton - 9/21-22)
- Peter MacDonald (Hamilton - 9/21-22)
- Steve Young (Hamilton - 9/21-22)
- Arianna Schlegel (Vassar - 9/21)
- Joanna DiPasquale (Vassar - 9/21-9/22)
- Francesca Livermore (Wesleyan - 9/21-22)
- John Neyssen (Wesleyan - 9/21-22)
- Lori Stethers (Wesleyan - 9/21-9/22)
- David Keiser-Clark (Williams - 9/21-22)
- Jessica Drmacich-Flach (Williams - 9/21 - virtual attendance)
- Jonathan Leamon (Williams - 9/21-22)
- Karen Gorss Benko (Williams - 9/21-22)
May 23-24, 2017: "ICG Hack/Doc" @ Williams College (Williamstown, MA)
- 8:30am Breakfast, get set up
- 9:30-10:30am Hack/doc begins! Introductions + planning
- CSV Importer
- Islandora-in-a-Box
- Notes for the day
- 10:30-10:35am Break
- 10:35-11:15am Break into hack/doc groups, check roles and goals
- 11:15am-12:15pm Hack + doc
- 12:15-1:15pm Lunch
- 1:15-3pm Hack + doc
- 3-3:15pm Break
- 3:15-4pm Hack + doc
- 4-5pm Preliminary reporting, discussion about what is going right and what isn’t working
- Post 5pm Dinner on your own (or we could get dinner together downtown if people are up for it)
- 8:30am Breakfast, get set up, notes for the day
- 9:30-10:30am Finish documenting and testing (code if necessary)
- 10:30-10:35am Break
- 10:35am- 12pm Informal presenting on the finished CSV Importer followed by informal presenting on recommended dev environments
- 12-1pm Lunch
- 1-3pm Discuss dev environment plan (to be sent to the Steering Committee for next steps/discussion)
- 3-3:15pm Break
- 3:15-4pm Wrap-up
- 4-5pm Reflection on what has been accomplished and goal setting for what comes next
- Post 5pm
Three projects/goals:
https://docs.google.com/document/d/1OqgkNnjeYoI_E_C5svk6fLqCO8La3jBR4XoDBcbzmBg/edit
CSV Importer
- The master CSV Importer is now back on the ICG github.
CSV Backend
CSV UI
- icig_csv_import_ui
For Day 2:
- Add issues from here https://docs.google.com/spreadsheets/d/1dLGhCSPThKrGlXX0Hu6X_ONPUqDPlvjiy5EgcQbTW3g/edit#gid=0
- To the issues queue here https://github.com/Islandora-Collaboration-icg_csv_import
- Sample csv with multiple attributes
- Write up install methods - Francesca & Gavin
- Git install
- Drush
Islandora in a Box
More info https://drive.google.com/drive/folders/0B--V-99YjZHNOEtuYmJVS2pEQ3M
Add to the stack matrix https://docs.google.com/a/wesleyan.edu/spreadsheets/d/1iM26Iv6s-T-ZApiftSZMKgqbIt7H-eOO0XU2oc7o4aw/edit?usp=drive_web
For Day 2:
-
Fill in the matrix (as possible)
-
There is a bash script - learn more about it.
-
1 hour sprint on metadata
-
Should there a working group?
-
More on the general sense of where this proposal stands
-
The Islandora-in-a-Box project’s goal is for the working group to return an improved Islandora-in-a-Box document to the ICG Steering Committee for their consideration, discussion and acceptance as a project for shared development. Steps to be done at this hack/doc include:
-
Intro to Islandora-in-a-Box concept and outline the steps we hope to accomplish at the hack/doc:
-
1 hour: whole group discussion: feedback, concerns and questions. Shake the tree. Improve the plan. Identify challenges and opportunities.
-
1 hour: sprint (partial group?) to question/understand/populate the "Stack Inventory Spreadsheet" (focus on collaboration)
-
1 hour: sprint (partial group?) to question/understand/create/populate a "Metadata Inventory and Guidelines Spreadsheet" (focus on collaboration)
-
Major concern: not just Islandora updates, but stack-level updates are cumbersome. Goal of shared level of confidence, reducing duplicate work across campuses, greater security.
Working committee could use dev environment approach to be a true production environment. Benefits include: stack maintenance streamlined; common core setup and maintenance with minimal cost.
Goal: get more people at the table to generate good discussion about how this will operate. From there, Mellon funding?
Major questions/discussions
-
Is this the right future-proof project?
-
Are there more narrow projects that we should focus on?
-
Will we alienate the rest of the Islandora community? (Is this a fork of the Islandora system?)
-
We’re trying to layer on what is already done.
-
We want to be sure that whatever we do could potentially be written into core, so we’d test against this
-
Big key: we’re thinking about process, not product
-
-
There is no "correct" Islandora stack → can we make headway here?
-
Model similar to Pantheon or Acquia (as examples) → we want to have a containerized system where we work with the application but not at the server/stack/OS level. Major undertaking, though, and would require us to be on board with a more centralized hosting solution.
-
Timing, 8.x vs 7.x.
-
Vagrant boxes for 7.x no longer really maintained, focused on dev env. Could we put in something production-worthy for our current needs?
-
Migration questions for current, mature repositories
-
-
Benefit for mature repositories?
-
Stability
-
Better migration to CLAW when the time comes
-
-
Data doubling problem: ability to generate sample data sets on development deploys [pat: contribute a core hook that content model modules can implement: hook_generate_samples()?]
-
Q: what is CLAW?
- D8 + nodes + Fedora 4 + microservices → tighter coupling with Drupal, less intense stack resources in order to do things like make derivatives (e.g., using separate server to make video derivatives with ffmpeg)
-
Could we go to shared instances?
-
Five College Compass: Digital Collections (Hampshire, Mount Holyoke and Smith Colleges) is doing this
-
Eventually → put on roadmap, but perhaps not now
-
In many ways, Williams and others are multi-tenant capable.
- Good reason to have the metadata / docker group and the coder group in the same room!
-
-
How "customized" can we already be to make this viable?
-
Goal for today to get the spreadsheet filled out, get a site survey
-
Could we accommodate all? (freedom + cost)
-
-
Big question: we’ve focused a lot on stack, less on application layer (another good focus for this group today/tomorrow)
-
Website/Github
Reports from Groups:
-
UI (Ben) - progress on UI
-
Multiple attributes (Mark) -
- Step one is working - multiple attributes in the xpath
-
ContentDM import (Peter) - working but no button yet, pulls in the JPEG2000 not the TIFF
-
Mark is the project lead. We will use the ICG github to manage issues and determine releases.
How to contribute to the repo: https://github.com/Islandora-Collaboration-Group/icg_csv_import/wiki/How-to-Contrbute-Code
Schedule a code sprint for a few weeks in the future to continue work
CSV Importer -
-
Add ICG_Hooks to the Islandora_Collaboration_Group github (from Digital Grinnell)
- FL with Pat Dunleavy after lunch!
-
Add a note to the documentation that this is also a necessary module
-
Mark - working on multiple attributes
- Trying to improve on the error reporting - you should get info in front of the AJAX error that is actually meaningful
-
UI update (Ben) - trying to go back a bit earlier in the XML form stream but not quite there yet. Inserting headers from CSV into the fields on the form builder,
-
Ajax is semi-functional--can add a duplicate element
-
Joanna wants to insert into the form exactly where you are and recursively work up the tree--so there is an explicit xpath that can be edited
-
Still doing development in the main csv importer, but will push it out to the UI module when it’s ready. Mark would like to start to split up the back and front-end code
Bash scripts for building the ICG matrix
-
Trying to build scripts that provides a simple set of information
-
https://github.com/Islandora-Collaboration-Group/islandora_stack_inventory
-
Testing as writing
-
Intentionally not creating custom messages for delivery of information, to make error reporting easier
-
People (who feel ready for it) can try it and add issues back into the repo
-
Nothing in the script reveals anything that could be used as an attack vector (no names, paths, passwords)
Structure of Github site
ICG_Information repo
-
Will house a hackdocs repo
- Daily notes and summary doc
-
Repo for the steering committee for meeting notes if they want to
-
Motion to consolidate all of our documentation, grow a sample set to use for testing (or even a hook to create sample data)
Mark and Pat
-
Latest changes propagated back to the ICG repo
-
More exception handling - XML creation of a MODS datastream, you will get a message somewhere with the root cause, particularly if it’s a problem with data
-
There is a lot of derivative and mods_to_DC code in the bottom of this process - replace lots of code with one little code
-
Fire all the hooks that all the content models implement to all of the derivative generation
-
Mark - exception handling, Islandora post-ingest/ingestion
Steve and John
- Continuing to refine the Bash scripts
Peter and Kevin
- Continuing to work on the documentation for the csv_importer
Joanna and Jonathan: UI
-
Working on getting xpaths into the interface
-
They’ve got a solution but it is doesn’t stop iterating/recursing
David and Karen
- GitHub
ICG Information repo
-
Stubbed out tree of where to put documentation
-
Hackdocs
-
Templates
-
ICG steering committee (blurb or point to notes)
-
Main readme page has a skeleton of a "real" information page for the group
-
David is going to keep working on this
-
Would like to turn off the Google site.
We should have a Metadata working group/Digital Humanities
We are struggling with scaling Islandora so that it fits individual solutions but doesn’t sacrifice enterprise level work/interoperability
You need to establish the rules/guidelines/whatever before you can break/exploit it
TO DO:
-
Recommendation to Steering Committee to form a Metadata working group that is cross-functional
-
Recommendation that a future hackdoc be focused on metadata/digital humanities
-
Get TODO issues into issue queue rather than a list
-
Islandora-in-a-box group needs to come up with a short term recommendation on dev environment
-
Mark will address the issues queue on the ICG_csv_import module
- Structured and repeatable tests possibly with Travis CI
-
Peter will work on the ContentDM button!
-
David to send paragraph to ICG: why IIAB is good
-
David to do github site edits
-
Working group will review the IIAB proposal and will bring it back to Steering Group
-
Decision: Ultimately - we would like the CSV_Importer to go to Islandora Labs (or maybe core) - maybe for unveiling at IslandoraCon