Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Assigning large files as datastream content fails #13

Open
cbeer opened this issue Mar 21, 2012 · 1 comment
Open

Assigning large files as datastream content fails #13

cbeer opened this issue Mar 21, 2012 · 1 comment

Comments

@cbeer
Copy link
Member

cbeer commented Mar 21, 2012

Large files (for some definition of large??) can't be read into Ruby, so they can't be added as object datastreams:

/Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:121:in `read': Invalid argument - HDS/barcode334930_1.mov (Errno::EINVAL)
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:121:in `content'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:143:in `attribute_change'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:117:in `block in changes'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:117:in `map'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activemodel-3.2.2/lib/active_model/dirty.rb:117:in `changes'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:245:in `to_api_params'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:210:in `block in create'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:403:in `_run__3645590890494944274__create__594327490570200131__callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `__run_callback'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:385:in `_run_create_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:209:in `create'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:221:in `block in save'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:403:in `_run__3645590890494944274__save__594327490570200131__callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:405:in `__run_callback'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:385:in `_run_save_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/activesupport-3.2.2/lib/active_support/callbacks.rb:81:in `run_callbacks'
    from /Users/chris_beer/.rvm/gems/ruby-1.9.3-p0@hydradam/gems/rubydora-0.5.7/lib/rubydora/datastream.rb:220:in `save'

See also:

1.9.3p0 :009 > f = open('HDS/barcode334930_1.mov')
 => #<File:HDS/barcode334930_1.mov> 
1.9.3p0 :010 > f.size
 => 3799020967 
1.9.3p0 :011 > f.read; 1
Errno::EINVAL: Invalid argument - HDS/barcode334930_1.mov
    from (irb):11:in `read'
    from (irb):11
    from /Users/chris_beer/.rvm/rubies/ruby-1.9.3-p0/bin/irb:16:in `<main>'

=== vs ====

1.9.3p0 :013 > f = open('/Users/chris_beer/Desktop/tmp')
 => #<File:/Users/chris_beer/Desktop/tmp> 
1.9.3p0 :014 > f.size
 => 3261 
1.9.3p0 :015 > f.read
 => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<xsl:stylesheet xmlns=\"http://www.pbcore.org/PBCore/PBCoreNamespace.html\" xmlns:xsl=\"http://www.w3.org/1999/XSL/Transform\" version=\"1.0\">\n    <xsl:output indent=\"yes\"/>\n    <xsl:strip-space elements=\"*\"/>\n    <xsl:template match=\"//ROW\">\n        <pbcoreDescriptionDocument xmlns=\"http://www.pbcore.org/PBCore/PBCoreNamespace.html\" schemaVersion=\"2.0\">\n            <pbcoreIdentifier>\n                <xsl:attribute name=\"source\">MARS Program Record ID</xsl:attribute>\n                <xsl:value-of select=\"ID_PROGRAM/text()\" />              \n            </pbcoreIdentifier>\n            <xsl:apply-templates select=\"*[normalize-space()]\"></xsl:apply-templates>\n        </pbcoreDescriptionDocument>\n    </xsl:template>\n                  \n    <xsl:template match=\"TITLE_SERIES\">\n        <pbcoreTitle>\n            <xsl:attribute name=\"titleType\">Series</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreTitle>\n    </xsl:template>    \n\n    <xsl:template match=\"TITLE_PROGRAM\">\n      <pbcoreTitle>\n        <xsl:attribute name=\"titleType\">Program</xsl:attribute>\n        <xsl:value-of select=\"text()\" />\n      </pbcoreTitle>\n    </xsl:template>\n\n    <xsl:template match=\"DATE_RELEASE\">\n        <pbcoreAssetDate>\n            <xsl:attribute name=\"dateType\">broadcast</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreAssetDate>\n    </xsl:template>\n\n    <xsl:template match=\"COVERAGE_DATE_PORTRAYED\">\n        <pbcoreCoverage>\n            <coverage>\n            <xsl:value-of select=\"text()\" />\n            </coverage>\n            <coverageType>temporal</coverageType>\n        </pbcoreCoverage>\n    </xsl:template>\n\n    \n    <xsl:template match=\"RIGHTS_SUMMARY\">\n        <pbcoreRightsSummary>\n            <rightsSummary>\n                <xsl:attribute name=\"annotation\">rights summary</xsl:attribute>\n                <xsl:attribute name=\"source\">WGBH MARS</xsl:attribute>\n                <xsl:value-of select=\"text()\"/>\n            </rightsSummary>\n        </pbcoreRightsSummary>\n    </xsl:template>\n\n    <xsl:template match=\"ID_NOLA\">\n        <pbcoreIdentifier>\n            <xsl:attribute name=\"source\">NOLA Code</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreIdentifier>\n    </xsl:template>\n    \n    <xsl:template match=\"SOURCE_PROGRAM_NUMBER\">\n        <pbcoreTitle> \n            <xsl:attribute name=\"titleType\">Episode</xsl:attribute>\n            <xsl:value-of select=\"text()\"/>\n        </pbcoreTitle>\n    </xsl:template>\n                   \n    <xsl:template match=\"DESCRIPTION_PROGRAM\">\n        <pbcoreDescription>\n            <xsl:attribute name=\"descriptionType\">Program</xsl:attribute>\n            <xsl:attribute name=\"descriptionTypeRef\">http://metadataregistry.org/concept/show/id/1702.html</xsl:attribute>\n            <xsl:value-of select=\"text()\" />\n        </pbcoreDescription>\n    </xsl:template>\n    \n    <xsl:template match=\"Producer\">\n        <pbcoreCreator>\n            <creator><xsl:value-of select=\"text()\"/></creator>\n            <creatorRole ref=\"http://metadataregistry.org/concept/show/id/1425.html\">Producer</creatorRole>\n        </pbcoreCreator>\n    </xsl:template>\n    \n    <xsl:template match=\"text()\" />\n</xsl:stylesheet>"
@ghost ghost assigned cbeer Mar 21, 2012
cbeer pushed a commit that referenced this issue Mar 21, 2012
first pass to fix issue #13 -- pass through IO-type objects that can'…
cff91ce 
…t be IO#read properly (because they are too large?)
cbeer pushed a commit that referenced this issue Mar 21, 2012
Revert "first pass to fix issue #13 -- pass through IO-type objects t…
89259a0 
…hat can't be IO#read properly (because they are too large?)"

This reverts commit cff91ce -- turns
out rest-client suffers from the same ruby feature, so this trickery
doesn't actually buy us anything after all?
@cbeer
Copy link
Member Author

cbeer commented Mar 21, 2012

See https://gist.github.com/2147407 for large file ingest experiments

@cbeer cbeer removed their assignment Aug 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@cbeer