Skip to content

Publishing New Volumes to Website

Jump to bottom
Joe Wicentowski edited this page Mar 7, 2024 · 42 revisions

NOTE: These steps assume that you have already received a final PDF for the volume that you have cropped and renamed according to FRUS file naming conventions. They also assume that you have an up-to-date history.state.gov Development Environment on your computer. See https://github.com/HistoryAtState/hsg-project/wiki/setup. Also, particular steps of the publishing process will require installing additional software packages, and the directions for those steps will lead you through the installation of these software packages as needed.

1. Edit Volume Metadata

In order to be able to create the shell file, you must first edit the metadata file for the volume and upload it to localhost.

Open the volume’s corresponding volume XML file in hsg-project/repos/frus/bibliography and copy and paste the first two paragraphs of the press release in <summary>. Since this file is not coded in TEI XML, when you tag paragraphs and text styling you will need to add the TEI prefix and namespace to each tag (ex. <tei:p xmlns:tei="http://www.tei-c.org/ns/1.0"> and <tei:hi rend="italic">). You also need to provide any missing information in the file (ISBN, General Editor, publication date, etc.) and correct information already there.

In order to view the volume metadata and shell file locally, you will need to change the volume’s <publication-status> to “published” and upload the file and the shell file you will create in step 2 to localhost. Here you can make corrections to the file and re-upload to localhost to view your corrections. Neither of these files should be uploaded to hsg, however, until you have completed all of the steps below and are ready to publish the volume.

Do you have the XML for the volume?

  • If YES, skip step 2 and proceed to step 3.

  • If NO, proceed to step 2.

2. Create Shell File

After you have uploaded the volume metadata file to the localhost, navigate to the Release app at http://localhost:8080/exist/apps/release/ and click on Shell File Helper. On the next page, enter the volume id and click Generate Shell Files. A draft shell file for the volume will download to your computer. Open the file in oXygen and save it in hsg-project/repos/frus/volumes.

Double check the information that is already populated in the shell file and fill out the div/@xml:id=”pressrelease” portion of the template with:

  • The publication date
  • The content paragraphs of the press release
  • GPO S/N
  • Editor information

When you are finished, upload the file to the localhost and review the volume landing page.

3. Review the Volume XML

When you receive the final XML for your volume, you will largely follow the same steps for reviewing the volume that we followed before we moved to GitHub:

  1. Create an "issue" in GitHub at https://github.com/HistoryAtState/frus/issues and copy and paste the review checklist (found at Reviewing FRUS TEI) into the issue.
  2. Rename the final XML file according to our file naming conventions, move the file to hsg-project/repos/frus/volumes, and upload it to localhost. You will have to replace the file that is currently there if you published the volume.
  3. Review the file following the instructions at Reviewing FRUS TEI for any clarification.
  4. When you have finished, commit the new file to GitHub and upload to hsg.

Do you have the XML for the volume?

  • If YES, proceed to step 4.

  • If NO, skip step 4 and proceed to step 5.

4. Generate TIFFs, PNGs, eBooks, and Metadata

NOTE: In order to generate TIFFs and PNGs you need several software packages: Imagemagick and ExifTool. If you have already installed these dependencies, skip the following step.

4a. INSTALL IMAGEMAGICK and EXIFTOOL

  1. In Terminal, enter brew install imagemagick to install Imagemagick.
  2. Finally, enter brew install exiftool to install the ExifTool utility.

4b. CREATE TIFFS, PNGS, and EXIF.XML

Before proceeding, be sure you have installed Imagemagick and Exiftool as described in the previous step.

If you have the final PDF of the volume, you can make TIFFs and PNGs while you wait for the final XML. To do this you will need to:

  1. Create a new folder on your desktop, say, frus-temp, and place the final PDF in the folder, ensuring that you have renamed it according to our FRUS file naming conventions to match the volume's ID.

  2. Go to Spotlight in the upper-right hand corner of your Mac and search for “Terminal”. Open Terminal.

  3. Type cd (followed by a space), then drag and drop the folder you created in step 1 onto the Terminal window, which fills in the complete path to this directory. (The result will be a command in your Terminal that looks something like cd /Users/joe/Desktop/frus-temp.) Then paste this entire line into Terminal (note: you'll probably need to scroll to get the entire line's worth of text):

    for file in *.pdf; do FOLDER=`basename -s .pdf $file`; mkdir $FOLDER; cd $FOLDER; mkdir tiff; mkdir medium; mkdir pdf; cd ..; mv $file "$FOLDER/pdf/"; done

    This will create a parent folder with 3 subfolders named “pdf”, “tiff”, and “medium”. Then paste this entire line in:

    for folder in $(find * -maxdepth 0 -type d ); do gs -dBATCH -dNOPAUSE -q -sDEVICE=tiffg4 -dUseCropBox -r600 "-sOutputFile=$folder/tiff/%04d.tif" "$folder/pdf/$folder.pdf"; done

    This splits the PDF into TIFFs, placing the TIFFs in the “tiff” folder.

    for folder in $(find * -maxdepth 0 -type d ); do mogrify -path "$folder/medium" -format png -density 72 "$folder/tiff/*.tif"; mogrify -resize 'x800' "$folder/medium/*.png"; done

    This transforms the TIFFs into screen resolution PNGs that are in the “medium” folder.

    This step takes significantly longer to complete so be sure that the process is done running before you proceed.

  4. Run the following command to save metadata about the images' sizes (be sure you've installed the exiftool tool as described above):

     for folder in $(find * -maxdepth 0 -type d ); do cd $folder; exiftool -X -r -IFD0:ImageWidth -IFD0:ImageHeight -PNG:ImageWidth -PNG:ImageHeight * > exif.xml; cd ..; done

  5. Finally, upload these files to S3 with the hsg-publish application:

  • Visit hsg-publish at http://hsg-prod-backend1.hsg:8080/exist/apps/hsg-publish.
  • Select Upload files to S3.
  • Select your volume.
  • Select the Choose Files button for each category of files: PDF, PNG, TIFF, and Miscellaneous (for exif.xml).
  • Select the Upload button.
  • If you upload any wrong files by mistake, email Joe and ask him to delete them for you.

Have you installed Calibre, Kindle, and Parallel?

4c. INSTALL PARALLEL, CALIBRE, AND KINDLE

You need several software packages to generate ebooks: Parallel, Calibre, and Kindle.

  1. In Terminal, enter brew install parallel to install Parallel.
  2. Then, enter brew install calibre to install Calibre.
  3. Finally, enter brew install kindle to install Kindle.

4d. CREATE EBOOKS

Once you have the final XML for the volume and have reviewed it thoroughly, you are ready to make ebooks.

  1. In order to create eBooks, you will need to first open the volume XML and bibliography XML files in oXygen. (These files should be found in hsg-project/repos/frus/volumes and hsg-project/repos/frus/bibliography respectively.)
  2. Using the external tools menu in oXygen, upload both of these files to localhost so that the latest copy is available to eXist: Upload current file to localhost
  3. Navigate to the Release app at http://localhost:8080/exist/apps/release/
  4. In the Release app, click eBook Batch Helper.
  • Enter the volume ID(s) in the "Volume IDs" input box (e.g. frus1977-80v12).
  • Set the radio button to generate Both epub and mobi-bound epubs.
  • Click Generate eBooks.
  1. In your Downloads folder, you will see a folder entitled “frus-ebooks-[date]” with subfolders “epub” “mobi” and “mobi-bound”.
  • If you see at the bottom of the eBook Batch Helper that the eBook batch conversion has successfully completed, open the “epub” subfolder in your Downloads folder.
  1. In the "epub" subfolder, you should see a .epub file named with the volume’s frus-id. Open this file and review it in Apple's Books application.
  • Verify the file is properly formatted:
    • Check the table of contents
    • Click the hyperlinks for random documents
    • Check headings
    • Check lists
    • Check tables
    • Check any images
    • etc.
  1. In the "mobi" subfolder, you should see a .mobi file named with the volume’s frus-id. Open this file and review it in Kindle. Review the Mobi as you did the EPUB:
  • Verify the file is properly formatted:
    • Check the table of contents.
    • Click the hyperlinks for random documents.
    • Check headings
    • Check lists
    • Check tables
    • Check any images
    • etc.
  1. When you have completed the review of the .epub and .mobi file, upload them to S3 with the hsg-publish application:
  • Visit hsg-publish at http://hsg-prod-backend1.hsg:8080/exist/apps/hsg-publish
  • Select Upload files to S3.
  • Select your volume.
  • Select the Choose Files button for each category of files: EPUB and Mobi.
  • Select the Upload button.
  • If you upload any wrong files by mistake, email Joe and ask him to delete them for you.

In order to ensure that links to the PDFs and ebooks appears on the volume's landing page and the ebooks page on hsg, you will need to use hsg-publish. Proceed to step 5.

5. Update Volume Cache

In earlier steps, we uploaded PDFs, etc. to S3 using hsg-publish. Uploading these assets to S3 alone will not cause the assets to appear on hsg—e.g., on the volume's landing page or the ebooks page. To appear on hsg, hsg has to “cache,” or store, some basic information about these resources (i.e., their name, size, and last modified date). When you're ready to publish a volume so its links appear on hsg:

  1. Visit hsg-publish at http://hsg-prod-backend1.hsg:8080/exist/apps/hsg-publish
  2. Select Browse HSG cache
  3. Select your volume
  4. Select Refresh the cache

Now, when you visit the volume's landing page or the ebooks page, the links to the volume's PDF and ebooks will appear.

6. Add Subject Tags for the Volume

For each volume, the compiler is required to supply subject tags that must be entered and uploaded to hsg for every newly published volume. When you have received the tags from the compiler, you should use the Tag Helper to put them in proper form:

  1. Navigate to https://history.state.gov/cms/apps/tags/tag-helper.xq and enter the username and password you received during your training.
  2. In the space provided, copy and paste the subject tags and press Search
  3. Below the entry box, you should see a report that indicates how many tags were found for the number of tags you provided and a list of space-delimited tags for inserting into a TEI document. Double check that all of your tags have been found. If they haven't, double check the spelling, etc. against the list of approved tags and run the search again after correcting any errors. When all of your tags have been found, open the frus-tags.xml file at /repos/tags/tagged-resources/frus. In the file:
  4. Scroll down to the bottom of the document and locate the place where your volume should be inserted in the list of volumes that are already published.
  5. Copy a <study> for another volume and paste it where your volume belongs.
  6. Revise the title and link information in this study to match your volume, and replace the existing tag/@id with the space-delimited tags you generated using Tag Helper above.
  7. When you have fully revised the new <study> entry for your volume in the file, save your changes and press upload-current-file-to-history.state.gov under your External Tools in oXygen.

7. Create Carousel Entry

As with our previous workflow, in order to update the carousel for the publication of a new volume or quarterly release, you will need to create a new entry and update the display order. The files you need are now located at hsg-project/carousel/display-order and hsg-project/carousel/entries.

  1. Open display-order.xml and add your new <topic-id> at the top of display-order, deleting the lowest <topic-id> entry so there are no more than 3 of these in <display-order>.
  2. As with the shell files, to create a new topic entry, you can either open the most recent topics entry, edit the information for your volume—remembering to change the id to one higher, and save the file as a new entry, or your can open a blank XML document and paste in the following to edit and save in the repo:
<?xml version="1.0" encoding="UTF-8"?>
<topic>
    <id>72</id>
    <type>publication</type>
    <title>Now Available: <em>Foreign Relations of the United States</em>, 1977–1980, Volume XXX,
        Public Diplomacy</title>
    <body>This volume documents the public diplomacy efforts of the Jimmy Carter administration. A
        major emphasis of the volume is the role the United States Information Agency played in the formulation and
        implementation of public diplomacy.</body>
    <link>/historicaldocuments/frus1977-80v30</link>
    <image>https://static.history.state.gov/frus/frus1977-80v30/covers/frus1977-80v30.jpg</image>
    <image-description>Book cover of <em>Foreign Relations of the United States</em>, 1977–1980, Volume XXX,
        Public Diplomacy</image-description>
    <image-height>600</image-height>
    <image-width>400</image-width>
    <image-source-information>Created by the Office of the Historian</image-source-information>
    <created-by>eckrothsl</created-by>
    <created-datetime>2016-06-07T07:20:59.72-04:00</created-datetime>
    <last-modified-by>eckrothsl</last-modified-by>
    <last-modified-datetime>2016-06-07T07:20:59.72-04:00</last-modified-datetime>
</topic>

Once you have finished editing both files, upload them to localhost, and review them one last time before uploading to hsg when you are are ready to publish the volume.

8. Commit all files to GitHub

After you have finished publishing your volume, the files in each of these repos need to be committed to GitHub. Remember to click Sync after you commit the files to push your changes to the master branch. You will have to do this 3 separate times for the following files in 3 repos:

  1. carousel
  • display-order
  • numbered entry for volume
  1. frus
  • bibliography
  • volumes
  1. tags/tagged resources
  • frus-tags
Clone this wiki locally