Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Streaming music - WIP #127

Merged
merged 31 commits into from Aug 16, 2019
Merged
Changes from 3 commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
17314eb
Make vorbis decoder usable for streaming.
and3md Jul 31, 2019
912b122
Added TStreamedSoundFile and TStreamedSoundOggVorbis.
and3md Jul 31, 2019
d372e26
The first draft of streaming sound buffers.
and3md Jul 31, 2019
74dbeda
Draft 2 of streaming sound: Multiple playback at the same time, Don't…
and3md Aug 4, 2019
32a92f9
Fixed checking dangle pointer when setBuffer() with TOpenALSoundBuff…
and3md Aug 4, 2019
75e2895
Proper release stream resources.
and3md Aug 4, 2019
9bc7ab4
Rewind streamed sound file, this is necessary for looping.
and3md Aug 5, 2019
865e0af
Music streaming: Sound looping support
and3md Aug 5, 2019
8571620
Fixed typo Soc -> Sox.
and3md Aug 5, 2019
961aa9c
TSoundBufferType -> TSoundLoading.
and3md Aug 5, 2019
8ec769f
Backward compatibility version of LoadBuffer.
and3md Aug 6, 2019
80c2989
Added CASTLE_SUPPORTS_THREADING, when threads are not available, fall…
and3md Aug 6, 2019
365eec4
Streaming: Code cleaning and minor changes.
and3md Aug 6, 2019
2b66e08
Remove unneeded log.
and3md Aug 11, 2019
b77bd60
Fix LoadBuffer result not assigned, do not introduce new LoadBuffer d…
michaliskambi Aug 12, 2019
f310a85
Whitespace (only one empty line to separate)
michaliskambi Aug 12, 2019
129102d
Wrap OggVorbis loading in TOggVorbisStream
michaliskambi Aug 12, 2019
823af29
Allow testing streaming in examples/audio/play_sounds
michaliskambi Aug 12, 2019
2f08594
Past tense of "read" is "read", not "readed"
michaliskambi Aug 13, 2019
a999ad6
Fix raising exception
michaliskambi Aug 13, 2019
853c13a
Every sound format (OggVorbis, WAV) is now expressed as TSoundReadEvent
michaliskambi Aug 13, 2019
23cf2ba
Remove large and unused TSoundFile.DataStatistics implementation
michaliskambi Aug 13, 2019
d0b8504
Allow loading FMOD dynamically (so we know at runtime when FMOD is no…
michaliskambi Aug 15, 2019
2c965d0
FMOD supports streaming too, and some code cleanups
michaliskambi Aug 15, 2019
71f4b63
Improve comment - Pascal threads for streaming are required by OpenAL…
michaliskambi Aug 15, 2019
8429af5
Fix FMOD dynamic freeing, to work with any finalization order of units
michaliskambi Aug 16, 2019
8ee8042
Get correct Duration for OggVorbis sounds, even when streaming
michaliskambi Aug 16, 2019
5315ea5
Various code cleanups (no functional changes)
michaliskambi Aug 16, 2019
4167622
Small comment improvements and code cleanups
michaliskambi Aug 16, 2019
8a06c75
Fix playing short sounds with slStreaming, we may need less than 4 bu…
michaliskambi Aug 16, 2019
dbf7c90
Improve comments and tiny code cleanups
michaliskambi Aug 16, 2019
File filter...
Filter file types
Jump to…
Jump to file or symbol
Failed to load files and symbols.

Always

Just for now

@@ -74,6 +74,23 @@ TSoundBufferBackendFromSoundFile = class(TSoundBufferBackend)
procedure ContextOpen(const AURL: String); override;
end;


{ TSoundBufferBackend descendant that can load TSoundFile instance.
Should be used by sound backends that cannot load sound files themselves,
and rely on TSoundFile to do it (right now, this applies to all backends except FMOD). }

{ TSoundBufferBackendFromStreamedFile }

TSoundBufferBackendFromStreamedFile = class(TSoundBufferBackend)
protected
{ Load from @link(SoundFile).
When overriding, call inherited first.
@raises Exception In case sound loading failed for any reason. }
procedure ContextOpenFromStreamedFile(const StreamedSoundFile: TStreamedSoundFile); virtual;
public
procedure ContextOpen(const AURL: String); override;
end;

{ Abstract sound engine sound source: something in 3D that plays sound. }
TSoundSourceBackend = class
strict private
@@ -137,7 +154,7 @@ TSoundEngineBackend = class
procedure ContextClose; virtual; abstract;

{ Create suitable non-abstract TSoundBufferBackend descendant. }
function CreateBuffer: TSoundBufferBackend; virtual; abstract;
function CreateBuffer(BufferType: TSoundBufferType): TSoundBufferBackend; virtual; abstract;

{ Create suitable non-abstract TSoundSourceBackend descendant. }
function CreateSource: TSoundSourceBackend; virtual; abstract;
@@ -160,6 +177,31 @@ implementation

uses SysUtils;

{ TSoundBufferBackendFromStreamedFile }

procedure TSoundBufferBackendFromStreamedFile.ContextOpenFromStreamedFile(const StreamedSoundFile: TStreamedSoundFile);
begin

end;

procedure TSoundBufferBackendFromStreamedFile.ContextOpen(const AURL: String);
var
F: TStreamedSoundFile;
begin
inherited;

F := TStreamedSoundFile.CreateFromFile(URL);
try
FDuration := -1;
FDataFormat := F.DataFormat;
FFrequency := F.Frequency;
ContextOpenFromStreamedFile(F);
except
FreeAndNil(F);
raise;
end;
end;

{ TSoundBufferBackend -------------------------------------------------------- }

constructor TSoundBufferBackend.Create(const ASoundEngine: TSoundEngineBackend);
@@ -21,7 +21,7 @@
interface

uses SysUtils, Classes,
CastleUtils, CastleTimeUtils, CastleSoundBase;
CastleUtils, CastleTimeUtils, CastleSoundBase, CastleInternalVorbisFile;

type
ESoundFileError = class(Exception);
@@ -90,7 +90,58 @@ TSoundFile = class
procedure ConvertTo16bit; virtual;
end;

{ TStreamedSoundFile }

This comment has been minimized.

Copy link
@michaliskambi

michaliskambi Jul 31, 2019

Member

TStreamedSoundFile (interface, implementation of TStreamedSoundFile.CreateFromFile) is in part a copy-and-paste of TSoundFile. I know you had a reason for it (one returns TStreamedSoundFile), but maybe we can make the code more shared (to avoid code duplication, and have everything "coded once"). E.g. introduce something like TAbstractSoundFile that is common ancestor of TStreamedSoundFile and TSoundFile, and allows to reuse code better, e.g. provides a skeleton of CreateFromFile that is shared by both TStreamedSoundFile and TSoundFile?

I'm not sure what is the best solution here (sorry -- falling asleep now :) ). I know that the current solution copies some code from TSoundFile, so this will make harder to maintain.

This comment has been minimized.

Copy link
@and3md

and3md Aug 6, 2019

Author Contributor

I was thinking about this and it's hard for me to create some reasonable base class, the use of which will not require continuous casting to the sub-type (as in the case of TAbstractSoundFile.CreateFromFile(...):TAbstractSoundFile). I think the differences in the method of use are too big for me to connect these two branches. A common ancestor will always require casting after createFromFile() or class responsibility will be unclean (by adding all methods and implement only some of them). Unless I do not see any obvious solution, that you see.

BTW. I know it's quite annoying to have two classes for each sound format like TSoundOggVorbis and TStreamedSoundOggVorbis. But I think it's easier to maintain two simple classes (with well-defined responsibility) than one with a lot of "if streming then" conditions.

This comment has been minimized.

Copy link
@michaliskambi

michaliskambi Aug 6, 2019

Member

I will need to think about it more :)

I understand the case you make. Indeed we want to have a clean code -- and I can see that sometimes (like in this case) it may be cleaner (producing code simpler to understand) to duplicate some stuff, than to try to reuse everything by a common class.

So, I think you're right here, and I'll try to do some thinking about it later :), maybe I'll be able to propose something.


TStreamedSoundFile = class
strict private
var
FURL: String;
public
{ Load a sound from a stream.
@param(URL Is used to initialize URL property.
It is not used to load, the URL contents are assumed to be in Stream.)
@raises(ESoundFileError If loading of this sound file failed.
E.g. in case of decoding problems
(e.g. we do not have vorbisfile / tremolo to decompress OggVorbis,
or the OggVorbis stream is invalid.)
)
@raises(EStreamError If case reading from the underlying stream failed
(e.g. strean ended prematurely).
The class function CreateFromFile will catch and reraise them
as ESoundFileError.
)
}
constructor CreateFromStream(const Stream: TStream; const AURL: String); virtual;

{ Load a sound data from a given URL.
@raises(ESoundFileError If loading of this sound file failed.
See @link(CreateFromStream) for various possible reasons.) }
class function CreateFromFile(const AURL: string): TStreamedSoundFile;

{ URL from which we loaded this sound file. }
property URL: String read FURL;

function DataFormat: TSoundDataFormat; virtual; abstract;
function Frequency: LongWord; virtual; abstract;

{ Returns readed size. }
function Read(var Buffer; const BufferSize: LongInt): LongInt; virtual; abstract;

{ Convert sound data to ensure it is 16bit (DataFormat is sfMono16 or sfStereo16,
not sfMono8 or sfStereo8).
The default implementation just raises an exception if data is not 16-bit.
When overriding this you call "inherited" at the end. }
procedure ConvertTo16bit; virtual;
end;

TSoundFileClass = class of TSoundFile;
TStreamedSoundFileClass = class of TStreamedSoundFile;

{ OggVorbis file loader.
Loads using libvorbisfile or tremolo. Both are open-source libraries
@@ -111,6 +162,24 @@ TSoundOggVorbis = class(TSoundFile)
function Frequency: LongWord; override;
end;

{ TStreamedSoundOggVorbis }

TStreamedSoundOggVorbis = class(TStreamedSoundFile)
private
FDataFormat: TSoundDataFormat;
FFrequency: LongWord;
SourceFileStream: TStream;
OggFile: TOggVorbis_File;
public
constructor CreateFromStream(const Stream: TStream; const AURL: String); override;
destructor Destroy; override;

function Read(var Buffer; const BufferSize: LongInt): LongInt; override;

function DataFormat: TSoundDataFormat; override;
function Frequency: LongWord; override;
end;

TSoundWAV = class(TSoundFile)
private
FData: Pointer;
@@ -134,9 +203,131 @@ TSoundWAV = class(TSoundFile)

implementation

uses CastleStringUtils, CastleInternalVorbisDecoder, CastleInternalVorbisFile,
uses CastleStringUtils, CastleInternalVorbisDecoder,
CastleLog, CastleDownload, CastleURIUtils;

{ TStreamedSoundOggVorbis }

constructor TStreamedSoundOggVorbis.CreateFromStream(const Stream: TStream; const AURL: String);
begin
SourceFileStream := Stream;
OpenVorbisFile(OggFile, SourceFileStream, FDataFormat, FFrequency);
end;

destructor TStreamedSoundOggVorbis.Destroy;
begin
CloseVorbisFile(OggFile);
FreeAndNil(SourceFileStream);

inherited Destroy;
end;

function TStreamedSoundOggVorbis.Read(var Buffer; const BufferSize: LongInt): LongInt;
begin
Result := ReadVorbisFileFillBuffer(OggFile, Buffer, BufferSize);
//Result := ReadVorbisFile(OggFile, Buffer, BufferSize);
end;

function TStreamedSoundOggVorbis.DataFormat: TSoundDataFormat;
begin
Result := FDataFormat;
end;

function TStreamedSoundOggVorbis.Frequency: LongWord;
begin
Result := FFrequency;
end;

{ TStreamedSoundFile }

constructor TStreamedSoundFile.CreateFromStream(const Stream: TStream;
const AURL: String);
begin
inherited Create;
FURL := AURL;
end;

class function TStreamedSoundFile.CreateFromFile(const AURL: string): TStreamedSoundFile;
var
C: TStreamedSoundFileClass;
S: TStream;
MimeType: string;
TimeStart: TCastleProfilerTime;
begin
TimeStart := Profiler.Start('Loading "' + URIDisplay(AURL) + '" (TStreamedSoundFile)');
try
try
{ soForceMemoryStream as current TSoundWAV and TSoundOggVorbis need seeking }
S := Download(AURL, [soForceMemoryStream], MimeType);
{ calculate class to read based on MimeType }
if MimeType = 'audio/x-wav' then
raise ESoundFileError.Create('TODO: Reading WAV sound files not supported. Convert them to OggVorbis.')
else
if MimeType = 'audio/ogg' then
C := TStreamedSoundOggVorbis
else
if MimeType = 'audio/mpeg' then
raise ESoundFileError.Create('TODO: Reading MP3 sound files not supported. Convert them to OggVorbis.')
else
begin
WritelnWarning('Audio', Format('Not recognized MIME type "%s" for sound file "%s", trying to load it as ogg', [MimeType, AURL]));
C := TStreamedSoundOggVorbis
end;

Result := C.CreateFromStream(S, AURL);

//Result.CheckCorrectness;

if LogSoundLoading then
begin
WritelnLog('Sound', 'Loaded "%s": %s, %s, frequency: %d', [
URIDisplay(AURL),
Result.ClassName,
DataFormatToStr(Result.DataFormat),
Result.Frequency
]);
{ This is informative, but takes some time, so is commented out.
WritelnLog('Sound', '"%s" data analysis: %s', [
URIDisplay(AURL),
Result.DataStatistics
]);
}
end;
except
{ May be raised by Download in case opening the underlying stream failed. }
on E: EFOpenError do
begin
S.Free;
{ Reraise as ESoundFileError, and add URL to exception message }
raise ESoundFileError.Create('Error while opening URL "' + URIDisplay(AURL) + '": ' + E.Message);
end;

{ May be raised by C.CreateFromStream. }
on E: EStreamError do
begin
S.Free;
{ Reraise as ESoundFileError, and add URL to exception message }
raise ESoundFileError.Create('Error while reading URL "' + URIDisplay(AURL) + '": ' + E.Message);
end;
on E: Exception do
begin
S.Free;
raise;
end;
end;

finally
Profiler.Stop(TimeStart)
end;
end;

procedure TStreamedSoundFile.ConvertTo16bit;
begin
// TODO: This could be implemented as independent from sound format
if not (DataFormat in [sfMono16, sfStereo16]) then
raise ESoundFileError.CreateFmt('Cannot convert this sound class to 16-bit: %s', [ClassName]);
end;

{ TSoundFile ----------------------------------------------------------------- }

class function TSoundFile.CreateFromFile(const AURL: string): TSoundFile;
@@ -80,7 +80,7 @@ TSoxSoundEngineBackend = class(TSoundEngineBackend)
public
function ContextOpen(const ADevice: String; out Information: String): Boolean; override;
procedure ContextClose; override;
function CreateBuffer: TSoundBufferBackend; override;
function CreateBuffer(BufferType: TSoundBufferType): TSoundBufferBackend; override;
function CreateSource: TSoundSourceBackend; override;
procedure SetGain(const Value: Single); override;
procedure SetDistanceModel(const Value: TSoundDistanceModel); override;
@@ -258,9 +258,14 @@ procedure TSoxSoundEngineBackend.SetListener(const Position, Direction, Up: TVec
begin
end;

function TSoxSoundEngineBackend.CreateBuffer: TSoundBufferBackend;
function TSoxSoundEngineBackend.CreateBuffer(BufferType: TSoundBufferType): TSoundBufferBackend;
begin
Result := TSoxSoundBufferBackend.Create(Self);
case BufferType of
sbtStreamed:
raise Exception.Create('Streamed buffers are not supported in Soc backend.');
This conversation was marked as resolved by and3md

This comment has been minimized.

Copy link
@michaliskambi

michaliskambi Jul 31, 2019

Member

Soc -> Sox :)

sbtFullLoad:
Result := TSoxSoundBufferBackend.Create(Self);
end;
end;

function TSoxSoundEngineBackend.CreateSource: TSoundSourceBackend;
@@ -64,6 +64,12 @@ TSoundDeviceList = class({$ifdef CASTLE_OBJFPC}specialize{$endif} TObjectList<
sfStereo16
);

TSoundBufferType = (
This conversation was marked as resolved by and3md

This comment has been minimized.

Copy link
@michaliskambi

michaliskambi Jul 31, 2019

Member

TSoundBufferType sounds a bit generic ("type of something" may mean various things), I would prefer

TSoundLoading = (
  slComplete,
  slStreaming
);
sbtFullLoad,
sbtStreamed
);


function DataFormatToStr(const DataFormat: TSoundDataFormat): string;

implementation
ProTip! Use n and p to navigate between commits in a pull request.
You can’t perform that action at this time.