Uploading of multimedia is extremely limited in medic-conf

[newtewt]

Webapp supports more media types than we can upload via medic-conf. Error shown when trying to upload mp3/mp4 is ERROR Error: Unrecongised file extension: mp3 for file /home/newt/dev/form-building/forms/app/media-examples-media/audio/audio.mp3

Looking at the code we only support the few types below.

switch(extension) {
    case 'json': return 'application/json';
    case 'png' : return 'image/png';
    case 'svg' : return 'image/svg+xml';
    case 'xml' : return 'application/xml';
    default: throw new Error(`Unrecongised file extension: ${extension} for file ${fileName}`);
  }

[garethbowen]

We should make sure the supported types are the same (eg: jpg, mp4, etc). I think we probably want to loosen this requirement so configurers can provide the mime type to get around this - there are simply too many file extensions and I don't want to have to list them all. Consider using a 3rd party library to resolve the mime type, eg: https://www.npmjs.com/package/mime-types

[garethbowen]

Ready for AT in medic-conf branch 332-support-mime-types. Verify that this works when uploading resource icons with a range of mime types (eg: png, svg, webp, jpg) and also embedding media in forms (eg: video, audio, images). Note that your browser must natively support the media.

[ngaruko]

I am seeing a long stack trace, can you have a look and see it it makes sense @mrsarm ? I have added a 112.4MB file to the resources folder.

INFO Actions:
     - upload-resources 
INFO Starting action: upload-resources… 

<--- Last few GCs --->

[3753:0x103fed000]     4740 ms: Scavenge 774.2 (809.6) -> 774.2 (809.6) MB, 99.9 / 0.0 ms  (average mu = 1.000, current mu = 1.000) allocation failure 
[3753:0x103fed000]     6315 ms: Mark-sweep 1542.2 (1577.6) -> 1520.9 (1556.1) MB, 594.8 / 0.0 ms  (+ 1.9 ms in 5 steps since start of marking, biggest step 0.5 ms, walltime since start of marking 5617 ms) (average mu = 0.899, current mu = 0.899) allocatio

<--- JS stacktrace --->
Cannot get stack trace in GC.
FATAL ERROR: MarkCompactCollector: semi-space copy, fallback in old gen Allocation failed - JavaScript heap out of memory
 1: 0x100ba1aee node::Abort() (.cold.1) [/usr/local/bin/node]
 2: 0x100084b65 node::FatalError(char const*, char const*) [/usr/local/bin/node]
 3: 0x100084c8d node::OnFatalError(char const*, char const*) [/usr/local/bin/node]
 4: 0x10017fcf5 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 5: 0x10017fc9f v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/bin/node]
 6: 0x100299e57 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/bin/node]
 7: 0x1002d20c6 v8::internal::EvacuateNewSpaceVisitor::Visit(v8::internal::HeapObject, int) [/usr/local/bin/node]
 8: 0x1002bfe83 void v8::internal::LiveObjectVisitor::VisitBlackObjectsNoFail<v8::internal::EvacuateNewSpaceVisitor, v8::internal::MajorNonAtomicMarkingState>(v8::internal::MemoryChunk*, v8::internal::MajorNonAtomicMarkingState*, v8::internal::EvacuateNewSpaceVisitor*, v8::internal::LiveObjectVisitor::IterationMode) [/usr/local/bin/node]
 9: 0x1002bf87a v8::internal::FullEvacuator::RawEvacuatePage(v8::internal::MemoryChunk*, long*) [/usr/local/bin/node]
10: 0x1002bf555 v8::internal::Evacuator::EvacuatePage(v8::internal::MemoryChunk*) [/usr/local/bin/node]
11: 0x1002d63d5 v8::internal::PageEvacuationTask::ProcessItems() [/usr/local/bin/node]
12: 0x1002d6370 v8::internal::PageEvacuationTask::RunInParallel(v8::internal::ItemParallelJob::Task::Runner) [/usr/local/bin/node]
13: 0x1002b3946 v8::internal::ItemParallelJob::Task::RunInternal() [/usr/local/bin/node]
14: 0x1002b3cd2 v8::internal::ItemParallelJob::Run() [/usr/local/bin/node]
15: 0x1002c1ed5 void v8::internal::MarkCompactCollectorBase::CreateAndExecuteEvacuationTasks<v8::internal::FullEvacuator, v8::internal::MarkCompactCollector>(v8::internal::MarkCompactCollector*, v8::internal::ItemParallelJob*, v8::internal::MigrationObserver*, long) [/usr/local/bin/node]
16: 0x1002c1c65 v8::internal::MarkCompactCollector::EvacuatePagesInParallel() [/usr/local/bin/node]
17: 0x1002b6d0b v8::internal::MarkCompactCollector::Evacuate() [/usr/local/bin/node]
18: 0x1002b4bb4 v8::internal::MarkCompactCollector::CollectGarbage() [/usr/local/bin/node]
19: 0x10029a135 v8::internal::Heap::MarkCompact() [/usr/local/bin/node]
20: 0x100298864 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/bin/node]
21: 0x100297853 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/bin/node]
22: 0x10029f2a0 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
23: 0x10029f2f6 v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/bin/node]
24: 0x10027c30d v8::internal::Factory::AllocateRawWithImmortalMap(int, v8::internal::AllocationType, v8::internal::Map, v8::internal::AllocationAlignment) [/usr/local/bin/node]
25: 0x10027e700 v8::internal::Factory::NewRawOneByteString(int, v8::internal::AllocationType) [/usr/local/bin/node]
26: 0x10054afdf v8::internal::IncrementalStringBuilder::Extend() [/usr/local/bin/node]
27: 0x1003578fc v8::internal::JsonStringifier::SerializeSmi(v8::internal::Smi) [/usr/local/bin/node]
28: 0x1003585e4 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
29: 0x100357d48 v8::internal::JsonStringifier::SerializeArrayLikeSlow(v8::internal::Handle<v8::internal::JSReceiver>, unsigned int, unsigned int) [/usr/local/bin/node]
30: 0x10035a805 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
31: 0x10035bf10 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
32: 0x10035bf10 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
33: 0x100357f7b v8::internal::JsonStringifier::SerializeJSReceiverSlow(v8::internal::Handle<v8::internal::JSReceiver>) [/usr/local/bin/node]
34: 0x10035b085 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<true>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
35: 0x1003596f4 v8::internal::JsonStringifier::Result v8::internal::JsonStringifier::Serialize_<false>(v8::internal::Handle<v8::internal::Object>, bool, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
36: 0x100354f71 v8::internal::JsonStringifier::Stringify(v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
37: 0x100354ebc v8::internal::JsonStringify(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/local/bin/node]
38: 0x1001fb113 v8::internal::Builtin_Impl_JsonStringify(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
39: 0x100750699 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_BuiltinExit [/usr/local/bin/node]
[1]    3753 abort      node ~/Medic/medic-dev/medic-conf/src/bin/medic-conf.js --local 

[garethbowen]

@ngaruko The key there is the "JavaScript heap out of memory" bit - I suspect nobody has tried with a 100+MB file before! We can probably get around that by streaming the file, but I think that's a separate issue. Can you raise an issue for that, and then test this with a smaller file?

[ngaruko]

@garethbowen Smaller size files are ok (different mime types)- raised a separate issue for this : #339

[mrsarm]

@ngaruko are we OK to move this ticket to Ready to merge?