Skip to content

Ipcamera for android

Jump to bottom Edit New page
Ricky Ng-Adam edited this page May 2, 2011 · 2 revisions

Introduction

References:

The problem it's trying to solve:

  • We want to stream video in real-time from an Android mobile phone camera to another host
  • Android expects a seekable file, waits for the recording to stop and then writes the header information

How is it solving the problem:

  • Creates a separate thread that receives the streamed camera data
  • inserts the necessary header information
  • pass along the information to a webserver
  • webserver sends video to the client (a Flash player)

Why it's interesting:

  • full source

Why it doesn't work as-is:

  • it expects a framerate of 10fps, much lower than the minimal 27 fps out of Nexus One
  • buffers and processing is all based on much smaller buffers

Checking out the source code:

  • Install, launch Eclipse
  • Install Subclipse (for Windows x64, you also need the x64 lib: https://www.sliksvn.com/en/download and add its bin directory to your path)
  • New Project > SVN Checkout > http://ipcamera-for-android.googlecode.com/svn/trunk
  • Project properties > Android 2.1 update 1
  • Connect your device (Nexus One or Haipad)
  • Right-click project > Run As > Android application
  • On your device, press start: will display an IP with port
  • from your PC, connect to that IP:port using your browser
  • Click the play button in the flash app
  • Camera recording on device starts
  • PROBLEM: player just keeps streaming and never shows picture

Console shows the following exceptions:

04-27 17:59:36.193: WARN/System.err(2604): java.net.SocketTimeoutException
04-27 17:59:36.193: WARN/System.err(2604):     at org.apache.harmony.luni.net.PlainSocketImpl.read(PlainSocketImpl.java:461)
04-27 17:59:36.193: WARN/System.err(2604):     at org.apache.harmony.luni.net.SocketInputStream.read(SocketInputStream.java:85)
04-27 17:59:36.193: WARN/System.err(2604):     at org.apache.harmony.luni.net.SocketInputStream.read(SocketInputStream.java:65)
04-27 17:59:36.193: WARN/System.err(2604):     at java.io.BufferedInputStream.fillbuf(BufferedInputStream.java:140)
04-27 17:59:36.193: WARN/System.err(2604):     at java.io.BufferedInputStream.read(BufferedInputStream.java:324)
04-27 17:59:36.193: WARN/System.err(2604):     at com.appdh.webcamera.Worker.handleClient(WebServer.java:169)
04-27 17:59:36.193: WARN/System.err(2604):     at com.appdh.webcamera.Worker.run(WebServer.java:126)
04-27 17:59:36.193: WARN/System.err(2604):     at java.lang.Thread.run(Thread.java:1019)
04-27 17:59:49.623: WARN/StagefrightRecorder(68): Intended video encoding frame rate (10 fps) is too small and will be set to (27 fps)

Outline of how it works

  • MainActivity
  • contains layout created with initLayout()
  • instantiates StreamingKernel, WebServer and MediaSource
  • onCreate
  • optionally loads a configuration file and sets mSetuped
  • creates a new MediaSource (the video camera)
  • on a button press, StartWork() is called
  • WebServer is instantiated
  • StreamingKernel is instantiated and a new thread for it is started
  • MediaSource is given the StreamingKernel file descriptor
  • MediaSource
  • wraps a MediaRecorder and a preview CameraView
  • outputs to MPEG_4 container with H264 data stream
  • frame rate set to 10, but actually 27
  • VGA resolution: 640x480
  • StreamingKernel
  • sets a constant "FRAME_SIZE"
  • creates n VideoBuffer to hold the incoming data ("Video Ring Buffer manager")
  • Each VideoBuffer instantiates a number of VideoPackage (originally 128) * each VideoPackage has data, size, flag, vflag and ts (timestamp)
  • instantiates a TimeStampEstimator
  • the main loop that is running in a separate thread
  • initiates streaming copy to file
  • skips the header (set to null by the recorder anyway...)
  • starts first frame timestamp tracking
  • in a loop * reads from source and fills buffer with per-frame 4 bytes * reads from that buffer frame information and determines package size * reads up to the size of frame information * in another loop to wait for the buffer ring to have availability
    • when there's availability in the VideoBuffer
    • write the current frame from the buffer
  • Issues?
  • receiver and send video size might be too small
  • set to frame size
  • commented out writing sample to file and duplicate code
  • dump directly whatever is written...
  • not closing fis consistently

On connection:

  • Client connects to WebServer
  • WebServer creates Worker
  • Worker calls MainActivity:doStreaming
  • doStreaming calls mMediaSource.startCapture() * mRecorder is started
    • mRecorder writes to the output file (read by the StreamingKernel thread)
    • StreamingKernel was waiting for data, discards the first 32 bytes from the recorder (bad header)
    • StreamingKernelto reads from mRecorder and writes "packages"
  • doStreaming writes FlvHeader, VideoHeader to the client
  • doStreaming reads "packages" from StreamingKernel video ring buffer to the client

video format

MPEG-4 container H.264/AVC video stream Baseline (8 bit sample depth, Flexible macroblock ordering (FMO), Arbitrary slice ordering (ASO), Redundant slices (RS)) L2.2 (4000Kbits/s)
Encoder: http://www.videolan.org/developers/x264.html

Android reported information

  • MPEG4Writer(4022): limits: 2147483647/0 bytes/us, bit rate: 192000 bps and the estimated moov size 3072 bytes
  • Sample file
  • total size: 92685 bytes for 2.836s
  • total data: 89589
  • MPEG header: 3096 bytes (0..3095)
  • mdat start header: 32 bytes (...mdat, 6d 64 71 74)
  • first frame size: 4 bytes (00 00 05 1f) = 1311 bytes

other tips

Comparing finalized video streaming with incomplete video stream:

dd if=test_playable.mp4  of=test_playable_same_size.mp4 bs=433420 conv=append count=1 skip=4372
cmp --verbose --print-bytes test_playable.mp4 test_unplayable.mp4
dd if=test_playable.mp4  of=combined.mp4 bs=4372 count=1
dd if=test_unplayable.mp4  of=test_playable_same_size.mp4 bs=433420 count=1 seek=4372

video reference

loong thread about streaming h.264 in mp4 or flv http://www.longtailvideo.com/support/forums/jw-player/servers-and-streaming/10253/stream-h264-with-php-script

explanation of how atoms work: http://atomicparsley.sourceforge.net/mpeg-4files.html http://wiki.multimedia.cx/index.php?title=MP4

good overview of what the atoms look like: http://atomicparsley.sourceforge.net/

FLV and F4V specification: http://read.pudn.com/downloads162/doc/fileformat/739073/video_file_format_spec_v9.pdf http://code.google.com/p/glyde-flv/wiki/FLVAnatomy

h264: http://stackoverflow.com/questions/3493742/problem-to-decode-h264-video-over-rtp-with-ffmpeg-libavcodec

THE H.264 SEQUENCE PARAMETER SET http://www.cardinalpeak.com/blog/?p=878

NAL format: http://stackoverflow.com/questions/1685494/what-does-this-h264-nal-header-mean +---------------+ |0|1|2|3|4|5|6|7| +-+-+-+-+-+-+-+-+ |F|NRI| Type | +---------------+

Example software:

http://iphome.hhi.de/suehring/tml/

4 bytes of size + 4 letters identifier

  • ftyp: type of file, versioning (sample 0x18 -> 24 bytes)
  • moov: 0x04f3 - > 1267 bytes
  • mvhd: 0x6c -> 108 bytes
  • trak:
  • tkhd:
  • mdia:
  • mdhd:
  • hdlr
  • minf
  • dinf
  • dref
  • stbl
  • stsd
  • avcl
  • stsc
  • stco
  • free (position 1291): 0x070d: 1805 bytes
  • (1291 + 1805= 3096)
  • mdat: 0x015df5: 89589
  • ???
  • NAL start (0x000001) : 0x1143 (4419)
    • 00 00 01 34 41 9a 02 05 82 04 75 6b ea d7 d5 be ad 5d 7a 4e be
    • 34: 100010:
    • F=1
    • NRI=0
    • Type=2

FLV format

  • FLV Header (9 bytes)
  • FLV file body
  • PreviousTagSize0 (4 bytes) = 0
  • Tag -> FLVTAG
  • TagType (1 bytes, VIDEODATA = 9)
  • DataSize (3 bytes, length of data in data field, sizeof(VIDEODATA))
  • Timestamp (3 bytes)
  • TimestampExtended (1 byte)
  • StreamID (3 bytes, always 0)
  • Data -> VIDEODATA * FrameType = 4 bit (1 (seekable AVC) or 2 (non-seekable AVC)) * CodecID = 4 bit (7 = AVC) * VideoData -> AVCVIDEOPACKET
    • AVCPACKETTYPE (1 byte, AVCNALU = 1)
    • CompositionTime (3 bytes)
    • Data (n bytes, content depending on AVCPACKETTYPE)
  • PreviousTagSize (4 bytes, sizeof FLVTAG)

building ffmpeg on Windows

Note: current git version is broken and needs a tiny patch to build

diff --git a/libavcodec/aaccoder.c b/libavcodec/aaccoder.c
index 9748fe1..fdb22ca 100644
--- a/libavcodec/aaccoder.c
+++ b/libavcodec/aaccoder.c
@@ -30,6 +30,7 @@
  * add sane pulse detection
  ***********************************/

+#include "libavutil/libm.h"
 #include <float.h>
 #include <math.h>
 #include "avcodec.h"
diff --git a/libavcodec/aacsbr.c b/libavcodec/aacsbr.c
index 6ac2cbc..afff693 100644
--- a/libavcodec/aacsbr.c
+++ b/libavcodec/aacsbr.c
@@ -32,6 +32,7 @@
 #include "aacsbrdata.h"
 #include "fft.h"
 #include "aacps.h"
+#include "libavutil/libm.h"

 #include <stdint.h>
 #include <float.h>

./configure --disable-doc --disable-yasm --enable-ffplay make

experimenting with container-less formats

temp.mp4 is a valid, captured video. dumping this:

ffmpeg -i temp.mp4 -f h264 -vcodec copy -y legit-raw

legit-raw now starts with 00 00 05 1f which was at offset 3104 in the original file, right after the mdat... in "streaming files" it's more like 44...

ffmpeg.exe -i valid.mp4 -vcodec copy -vbsf h264_mp4toannexb -an of.h264

ffmpeg -f flv -s 640x480 -vcodec h264 -i doStreaming.flv -f mp4 doStreaming.mp4

Add a custom footer
Add a custom sidebar

Clone this wiki locally