March 31-April 01, 2016
- CSV Importer
- Benjamin Rosser (Barnard - 3/31)
- Pat Dunlavey (CMI - 3/31)
- Noah Smith (CMI - 3/31)
- Mark McFate (Grinnell - virtual attendance)
- Greg Lord (Hamilton DHi - 3/31, 4/01)
- Lisa McFall (Hamilton DHi - virtual attendance)
- Peter MacDonald (Hamilton DHi - 3/31, 4/01)
- Steve Young (Hamilton DHi - 3/31, 4/01)
- Joanna DiPasquale (Vassar - 3/31, 4/01)
- Sharyn Cadogan (Vassar - 3/31, 4/01)
- Francesca Livermore (Wesleyan - 3/31, 4/01)
- David Keiser-Clark (Williams - 3/31, 4/01)
- Jessika Drmacich-Flach (Williams - 3/31, 4/01)
- Jonathan Morgan-Leamon (Williams - 3/31, 4/01)
September 21-22, 2016: "ICG Hack/Doc" @ Wesleyan University (Middleton, CT)
- We'll provide breakfast/refreshments and lunch
- 8:30am Breakfast, getting set up
- 9:30-10:30am Hack/doc begins! Introductions and skill sharing, review of Islandora
- 10:30-10:35am Break
- 10:35am - 11:15am Dividing into teams, scoping for each project
- 11:15am - 12:15pm Code/doc starts
- 12:15pm - 1:15pm Lunch! Welcome to people arriving for the afternoon
- 1:15pm - 3:00pm Coding, documentation, preliminary testing
- 3:00pm - 3:15pm Break (coffee will be available)
- 3:15pm - 4:30pm Coding, documentation, preliminary testing
- 4:30pm - 5:00pm Preliminary reporting, discussion about what is going right and what isn't working
- Post-5:00pm Dinner on your own
- 8:30am Breakfast, getting set up
- 9:30-10:30am Coding, documentation, testing
- 10:30-10:35am Break
- 10:35am - 11:15am Coding, documentation, testing
- 11:15 a.m - 11:35am Presentations (10 minutes for each group)
- 11:35am - noon Reflection, what went right and what we could have done better
- Noon For those staying around a bit before heading home, let Joanna know! I'm happy to coordinate lunch.
Pat Dunlavey, Common Media
-
Islandora looks different to each of us because we all have different perspectives
-
Most of what we’ll be doing today is at Drupal implementation level, but we’ll lend our perspectives to what we can achieve
What is Islandora?
Islandora is a bunch of Drupal modules that connects Drupal to Fedora.
Fedora lets us store digital assets (huge files directory!) with a method for finding things.
→ Ingest into Fedora
→ SPARQL query in the triplestore (storage model) -- predicate, verb, object to describe what it wants to find
→ Fedora "I got it!" Sends file back up
→ Islandora Tuque library that talks to fedora and manages this process
Solr searching capability and GSearch
-
Ingest into Fedora → GSearch triggers ingest of the metadata so it’s in the search engine
-
Solr is a good middle man here! :)
For every solution pack, there are lots of dependencies that need to exist on the server (e.g., to process video, make derivatives, etc.)
Every object in Fedora consists of:
-
Core metadata
-
Datastreams
-
Different objects have different datastreams
In terms of developing, we’re basically talking about Drupal development
-
Drupal standards
-
Drupal methods
- Hooks, API, division
What is a hook?
-
When Drupal loads, there is a series of steps (initialization, etc.)
-
Mymodule_init → fires during initialization phase
-
All of Drupal operates on this principle! Expose hooks at various times, implement it in your module, then hook will fire during Drupal processing.
Two module files:
-
Modulename.info → title, description, dependencies
-
Modulename.module → just implements hooks (BEST PRACTICE)
-
If you want to write big, elaborate functions, then write in a separate file and use a module_load_include declaration.
- Optimization: Drupal then doesn’t have to load the big, elaborate code until it’s necessary :)
-
Islandora tends to put things in an /includes/ directory.
We all come from different vantage points, and we are all skilled in different ways. So it’s great to all be together!
Key: in addition to being a software stack, it’s also a community. We need to understand the software and how the community works. Can’t do things in isolation! Want to contribute, build relationships with other people, and call on them for help when needed.
-
Leverage platform
-
Leverage standards to contribute
-
Leverage standards to be able to help one another
-
Islandora / Fedora / Drupal communities -- need to be embraced :)
Good lesson: how do things get fixed in Islandora?
-
Open a Jira account (do we all have Jira accounts??? check!)
-
Look through existing issues -- did someone else also report this issue? Can watch, volunteer time, comment on it, etc.
-
Then, how to report:
-
Look at module
-
In own Github account, **fork **the upstream version of the module (make a local copy of it for yourself in GitHub)
-
Now, in your own development environment, clone your fork, make commits back upstream
-
Push back to your own fork: git put origin [branch]
-
Create pull request in GitHub
-
In Jira, comment that pull request was created
-
Then community will review, test, etc.
-
Auto testing (Travis.ci) happens when you do this
-
Will also test to see if you’re up to standards (even down to whitespace!)
-
-
-
**Thursdays at 2 p.m. EST **-- committers group meets on Skype -- agenda is a list of all pull requests and new Jira issues
-
If things look good, there will be a merge into HEAD (bleeding edge code)
-
If needs more work, volunteer to test
-
Sometimes things just don’t get into merging -- need testers, etc. -- community effort
-
Group is extremely welcoming!
-
Questions:
How do we future-proof?
-
This is always an issue in OSS -- things are always moving. We’re never going to get away from that.
-
But we need to be aware of what will be going away and what’s on the horizon.
-
Code longevity -- Fedora is the most stable piece in this ecosystem. It’s the institutional rock!
-
But there’s also the notion that the way that we connect might change, but maybe some of these methods will be solved so will be easy.
How do we get involved in the committers group?
-
Email Melissa Anez
-
There are also special interests groups, all volunteer efforts
Other ways to get assistance, chat, etc.:
-
Google groups: Islandora group, Islandora dev group
-
Islandora CLAW group -- Islandora 7.x-2.x + Fedora 4 -- Wednesdays
-
Diego Pino -- hosting 9 lessons on new stack -- http://connect.scholarsportal.info/islandora
-
Week One: Intro to Fedora 4.x(link is external)
-
Week Two: Hands-on Creating Fedora 4.x Resources(link is external)
-
Week Three: Data Flow in the CLAW(link is external)
-
Week Four: Intro to Drupal Side and some Camel
-
-
Two closed groups: Islandora Board of Directors, Islandora Roadmap Committee (review practices at the level of the Foundation, contributed modules that are contributed to the foundation)
We should think about how much we’re creating a concept as much as code!
Free open source software (FOSS) -- free as in kittens, not free as in beer. :)
-
We have our need to serve digital assets to the public, and there are lots of different ways to do this.
-
FOSS -- cost comes in as you use it as you need to develop, improve, even learn to use the software. These are all costs!
-
We’re making an investment in ourselves, in the community, etc.
-
If things don’t work the way we want, we need to realize that these are our problems, and this is how we solve them -- with our own investment, time, resources!
Day 2: Reports from Groups (April 1, 2016)
Overview of group work
Forms UI
-
csv headers associated with xpaths
-
decided to skip "save templates"
-
who is actually taking the csv loading it
-
Simple use case- metadata person will do it
-
Hardest use case- a faculty maps the fields to xpaths
-
Template is already built by specialist
-
We need a privileges for "okaying" the ingest
-
Agreement on spreadsheet, location of collection in Islandora, provided an excel, and give go-ahead
-
A place to review submission in entirety
-
Simple use case
FORMS UI
-
Pictures of board from yesterday.
-
Removing the ability to save templates
-
How does the mapping happen?
-
Backend -- CSV headers are associated with xpath. Gives ability to associate with MODS fields + attributes, multiple elements, etc., plus nested fields.
-
[+] or [-] more fields possibilities -- two auto-populate fields + any attributes you want to add. If nested field, can go in and add more.
-
-
Big question: what is our audience?
-
Big question: what is our workflow?
-
How much verification?
-
Could the verification be in a queue?
-
Could the upload go into a queue? For phase 2
-
Could the objects go into a queue? For phase 2
-
-
What are the params if we hide things to be very simple?
-
What content model
-
What form you’re directing into
-
Wanting to add:
Metadata review setup / queue Phase 2
Forms- Creation hack side
-
Most of the time looking for functions
-
Pat’s ICG Form
-
Import button
-
A pull from content models, creating a rendered html list
-
Multi-step form
-
It is working bug free
-
Multiple steps
-
More to do! It’s complicated!
-
No functionality
-
Needs a form submit button, which will drive the form
-
It will not get done today
CSV
-
everything in github
-
Load data and parse data
-
Using library to make it happen
-
Removes special character
-
Allow multiple values in csv to be parsed out
-
Working with test data
-
Ready to use
-
Their team is ready to help with other groups
Batch
-
Difficult to take xpath and generate an element
-
Mark M. goes into xml form that generates one form and uses Oxygen to find xpaths
-
Created a template based on xpaths, clones elements, and takes resulting document (removes xpath) and the data is left
-
Pat and Mark will work through this, later today If Mark needs other format
-
Wants to generate new csv to show what was imported
-
Creates a spreadsheet with all xpaths
-
Left of spreadsheet creates column with PIDs
-
Cell A1 tells whether the data contains processed information
-
Mark will document (Julia will help)
-
Mark M. needs assistant to test via simple test. Does anyone have experience?Can they look at the code. Pat will help!
-
Having the csv with xpaths and pids would be helpful (Joanna). Jessika agrees!
-
Mark M. says it is easy to do (creating the pid and xpaths) YES!
TESTING (Steve, Jessika, Peter)
-
Looked for good/bad data from a variety of collections
-
Looked for legacy data
-
Uploaded materials to GitHub for testing
-
Examples of MODS files that we are able to generate
-
README.md file
-
Standard headings, etc., that Islandora recommends
-
Populated with whatever we knew
-
Where are we going?/ Next steps?
We are not going to have a functional released model. What we need to do is: find someone to take the project on.
Can it be finished?
-
We learned to collaboratively develop code
-
We need to gather again for another hackdoc
-
Is this possible?
-
Suggestion: it needs to be an all-day virtual meeting
-
We need to figure out how to work virtually
-
Webex is great for recording for those who cannot attend
-
Look into better software or place for this to happen
-
-
Could we all work in a shared development environment? Google-docs for code
Let’s do another hack/doc.
6 months for another hack/doc (Wesleyan?)
Individual meetups
Additional people
Finish this code
Make a vagrant that puts together all of our work, scripts, etc.
It would be great to have a shared environment (e.g., the Common Media proposal) for coding, testing, development, etc. Canonical production platform that we would use for development.
Two next steps:
Do another hackdoc!
How do we get a shared development environment
Who do we bring in for hackdoc?
-
Are people from other institutions invited?
-
Someone from DGI?
-
Anyone who is in the NE who uses Islandora (including Grinnell).
Think about other opportunities to meet up
Mark M. "taking baton!" To finish project
-
Where are the pieces coming from
-
Who is involved
-
Reach out one-on-one effectively
On March 31, participants from Barnard, Grinnell, Hamilton, Wesleyan, Williams, and Vassar met at Vassar College for our first hack/doc. A hack/doc combines the form of a hackathon -- bringing coders together to work on a predetermined problem in a large block of time -- with a substantial documentation component. This is a much more inclusive and open way to conduct a hackathon, incorporating multiple perspectives and insights into the problems at hand.
-
I created a series of documents to prepare for the day, including a schedule, attendance list, and "goals and roles." The roles document, in particular, was most important, because it laid the foundation for the types of participation that each attendee could do.
-
The "Roles" document was pre-filled by attendees, allowing us to establish particular skill sets and resource points.
-
The "Roles" document also provided questions for us to decide: what type of hack/doc goals did we want? What should our focus be?
-
A few emails were exchanged among prospective attendees to define the coding project(s) to be worked on during the hack/doc. Nothing was decided, but it got the mental juices flowing.
-
We reviewed the documents and tasks from our meeting at Williams College.
-
We determined that we wanted our goals to focus on learning how to work together, even if it meant foregoing one of our two coding projects.
-
We removed the project to improve the Compound Object Solution Pack template in favor of creating a new module, a CSV importer.
-
Improvement of the Compound Object Solution Pack template is desired but closely connected to the current Islandora version and may not be future-proof.
-
A CSV importer is sought-after functionality by the entire Islandora community and has immediate use for each ICG participant organization.
-
CSV importers simplify very complex procedures, so this code will take time and resources. Sharing this load is an ideal scenario for ICG.
-
-
Pat Dunlavey from Common Media provided an Islandora overview. It was fantastic!
-
We divided into teams to hash out functionality, wireframes, and goals for the module, then began coding, documenting, and sharing real-world examples of data.
-
We all got direct experience with GitHub: forking, cloning, committing, and creating pull requests on our shared repo and on our personal repos.
-
Our entire session was recorded and available online.
-
We reviewed Day 1’s work and setup to discover what was working and what was not. In particular, virtual participants had a difficult time following along what was happening in the face-to-face session.
-
We committed code to GitHub and continued working and documenting.
-
Mark McFate from Grinnell College kindly offered to be the point person for future coordination and updates to the module once we submit our work for consideration in Islandora Labs or more. (Participants recognized that this functionality is highly desirable in an Islandora installation and could perhaps be widely used, so it is good to have a succession plan/point person well in advance.)
-
We learned to collaboratively develop code. We can’t stress this enough.
-
The hack/doc concept! It was amazing to have so many different perspectives and skills in one room. In a short time, we were able to work out use cases, find real-world data, write some code, and connect with one another.
-
Roles. Without these, it would have been difficult to see how the concept was supposed to work and how we all came to the table as equal partners.
-
GitHub. Though we are somewhat "locked into" this code-sharing and versioning platform as Islandora users, it was nice to see that it fit fairly well into our workflow (and vice versa).
-
Participants. It was wonderful to work with colleagues from so many colleges and learn from one another!
-
It was very difficult for virtual participants to participate fully. We need to consider alternate platforms (WebEx? Adobe Connect?) or potentially alternate schedules (less whole-room participation and more coding groups is private rooms with remote connection capabilities).
-
We need to consider how to make our next / follow-up meetings virtual using Google Hangouts, WebEx, or some other similar platform. This is the only way that we will all be able to meet as frequently as our code and documentation development will demand.
-
A shared development environment is ideal. While working with virtual machines and pushing code up to GitHub provided a fairly smooth workflow, it made initial setup difficult because we did not have a ready-to-use environment to connect into.
-
Think about other avenues for meetups (e.g., Digital Scholarship and Liberal Arts conference at Macalester; DLF-LAC meeting in Milwaukee). Send email to group with events you’re attending!
-
Consider inviting other local Islandora orgs whenever there is a hack/doc taking place -- e.g., METRO-NYC, UConn, Baseball Hall of Fame. Potentially bring in DGI consultant as well.
-
Mark will continue code development and will coordinate with coders and documenters to finish project.
-
There is significant interest in another hack/doc within the next 6 months. Francesca (Wesleyan) has volunteered to coordinate.
-
Joanna will assemble basic costs for event so that other institutions have ballpark expectations. Planning documents are available in our shared ICG Google Drive folder. This should be an event that is easily replicable across our campuses.
-
A Vagrant environment that is inclusive of our own OS/setups.
-
A canonical production platform that we would use for development (e.g., Common Media proposal).
-
Membership in ICG -- others will be interested in our work -- where do we go from here?
Submitted by Joanna DiPasquale, Vassar College, April 4, 2016
Subsequently updated by Mark McFate and Peter MacDonald.