Add Audio API #1974

xyang16 · 2022-08-29T16:08:26Z

Description

Brief description of what this PR is about

Add Audio API

If this change is a backward incompatible change, why must this change be made?
Interesting edge cases to note here

frankfliu · 2022-08-29T16:11:11Z

api/src/main/java/ai/djl/util/Float16Utils.java

+     * @param bigEndian whether the data is stored in big-endian or little-endian order.
+     * @return a float array.
+     */
+    public static float[] fromByteArray(byte[] bytes, boolean bigEndian) {


Suggested change

public static float[] fromByteArray(byte[] bytes, boolean bigEndian) {

public static float[] fromByteArray(byte[] bytes, ByteOrder endian) {

frankfliu · 2022-08-29T16:13:03Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+
+    private static final Logger logger = LoggerFactory.getLogger(AudioFactory.class);
+
+    @SuppressWarnings({"PMD.DoubleBraceInitialization", "PMD.UseConcurrentHashMap"})


better avoid create a new anonymous class, you can use a static initializer instead.

This can be just a String[], no need use a map here

frankfliu · 2022-08-29T16:16:51Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+     * @return {@link AudioFactory}
+     */
+    public static AudioFactory newInstance(Configuration conf) {
+        boolean android = "http://www.android.com/".equals(System.getProperty("java.vendor.url"));


I think we should just a simple try and fail approach for both Android and PC

frankfliu · 2022-08-29T16:18:43Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+     * @param conf The configuration for {@link AudioFactory} to use.
+     * @return {@link AudioFactory}
+     */
+    public static AudioFactory newInstance(Configuration conf) {


I think we need newInstace() overload

frankfliu · 2022-08-29T16:20:58Z

api/src/main/java/ai/djl/modality/audio/Audio.java

+ * {@code Audio} is a container of an audio in DJL. The raw data of the audio is wrapped in a float
+ * array.
+ */
+public interface Audio {


Audio is just a data holder now. I'm not sure if we need a interface now.

frankfliu · 2022-08-29T16:21:18Z

api/src/main/java/ai/djl/modality/audio/Audio.java

+     * @param type type of the audio, such as "wav"
+     * @throws IOException audio cannot be saved through output stream
+     */
+    void save(OutputStream os, String type) throws IOException;


Let remove this since we didn't have any implementation now

frankfliu · 2022-08-29T16:22:21Z

api/src/main/java/ai/djl/modality/audio/SampledAudio.java

+
+    /** {@inheritDoc} */
+    @Override
+    public void save(OutputStream os, String type) throws IOException {


We can move save into Factory in needed in future

frankfliu · 2022-08-29T16:23:13Z

api/src/main/java/ai/djl/util/AudioUtils.java

+     * @return the bytes read from the audio input stream.
+     * @throws IOException if an input or output error occurs
+     */
+    public static byte[] read(AudioInputStream ais) throws IOException {


Should this implementation in SampleAudioFactory?

frankfliu · 2022-08-29T16:24:30Z

extensions/audio/src/main/java/ai/djl/audio/dataset/AudioData.java

-        int i = 0;
-        for (Float f : list) {
-            floatArray[i++] = (f != null ? f : Float.NaN);
+        for (int i = 0; i < list.size(); i++) {


I think old one is better:

doesn't need to call list.size() every thime

it works better on non-randomaccess lists (Like LinkedList)

frankfliu · 2022-08-29T16:26:21Z

extensions/audio/src/main/java/ai/djl/audio/util/FFmpegUtils.java

+     * @return the float array read from the audio.
+     * @throws FFmpegFrameGrabber.Exception if error occurs
+     */
+    public static float[] grab(FFmpegFrameGrabber grabber) throws FFmpegFrameGrabber.Exception {


This should be in FFmpegAudioFactory

codecov-commenter · 2022-08-29T16:34:15Z

Codecov Report

Merging #1974 (4d0dea2) into master (bb5073f) will decrease coverage by 1.93%.
The diff coverage is 68.64%.

@@             Coverage Diff              @@
##             master    #1974      +/-   ##
============================================
- Coverage     72.08%   70.14%   -1.94%     
- Complexity     5126     5859     +733     
============================================
  Files           473      578     +105     
  Lines         21970    25946    +3976     
  Branches       2351     2795     +444     
============================================
+ Hits          15838    18201    +2363     
- Misses         4925     6373    +1448     
- Partials       1207     1372     +165

Impacted Files	Coverage Δ
api/src/main/java/ai/djl/modality/cv/Image.java	`69.23% <ø> (-4.11%)`	⬇️
...rc/main/java/ai/djl/modality/cv/MultiBoxPrior.java	`76.00% <ø> (ø)`
...rc/main/java/ai/djl/modality/cv/output/Joints.java	`71.42% <ø> (ø)`
.../main/java/ai/djl/modality/cv/output/Landmark.java	`100.00% <ø> (ø)`
...main/java/ai/djl/modality/cv/output/Rectangle.java	`72.41% <0.00%> (ø)`
...i/djl/modality/cv/translator/BigGANTranslator.java	`21.42% <0.00%> (-5.24%)`	⬇️
...odality/cv/translator/BigGANTranslatorFactory.java	`33.33% <0.00%> (+8.33%)`	⬆️
...nslator/InstanceSegmentationTranslatorFactory.java	`14.28% <0.00%> (-3.90%)`	⬇️
.../cv/translator/SemanticSegmentationTranslator.java	`0.00% <0.00%> (ø)`
.../cv/translator/StyleTransferTranslatorFactory.java	`40.00% <ø> (ø)`
... and 480 more

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

frankfliu · 2022-08-29T18:45:45Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+    }
+
+    /**
+     * Create new instance of audio factory from the provided factory implementation.


Suggested change

* Create new instance of audio factory from the provided factory implementation.

* Creates new instance of audio factory from the provided factory implementation.

frankfliu · 2022-08-29T18:45:57Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+    }
+
+    /**
+     * Gets {@link Audio} from file.


Suggested change

* Gets {@link Audio} from file.

* Returns {@link Audio} from file.

frankfliu · 2022-08-29T18:46:10Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+        }
+
+        /**
+         * Construct a new {@code Configuration} instance.


Suggested change

* Construct a new {@code Configuration} instance.

* Constructs a new {@code Configuration} instance.

frankfliu · 2022-08-29T18:46:45Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+    public static final class Configuration {
+
+        /** Default sample rate. */
+        public static final int DEFAULT_SAMPLE_RATE = 16000;


Does this need to be public?

Changed to private.

frankfliu · 2022-08-29T18:47:07Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+        }
+
+        /**
+         * Get the number of channels of an audio file.


Suggested change

* Get the number of channels of an audio file.

* Returns the number of channels of an audio file.

frankfliu · 2022-08-29T18:47:16Z

api/src/main/java/ai/djl/modality/audio/AudioFactory.java

+        }
+
+        /**
+         * Set the number of channels for {@link AudioFactory} to use.


Suggested change

* Set the number of channels for {@link AudioFactory} to use.

* Sets the number of channels for {@link AudioFactory} to use.

frankfliu · 2022-08-29T18:47:56Z

api/src/main/java/ai/djl/util/Float16Utils.java

+     * @param endian whether the data is stored in big-endian or little-endian order.
+     * @return a float array.
+     */
+    public static float[] fromByteArray(byte[] bytes, boolean endian) {


Suggested change

public static float[] fromByteArray(byte[] bytes, boolean endian) {

public static float[] fromByteArray(byte[] bytes, ByteOrder endian) {

There actually 3 endian type:
BIG, LITTLE and NATIVE

https://docs.oracle.com/javase/8/docs/api/java/nio/ByteBuffer.html#order-java.nio.ByteOrder-

bo - The new byte order, either BIG_ENDIAN or LITTLE_ENDIAN.

NATIVE is actually BIG_ENDIAN or LITTLE_ENDIAN

Also I got whether it's big-endian or little-endian from AudioFormat, so there's no native involved.

https://github.com/xyang16/djl/blob/audio/api/src/main/java/ai/djl/modality/audio/SampledAudioFactory.java#L52

float[] floats = Float16Utils.fromByteArray(bytes, format.isBigEndian());

You can choose to only pass in BIG/LITTLE to this API in AudioFactory, but the API should be able to accept any endian. The source of the input to this API may not only come from AudioSystem

frankfliu · 2022-08-29T20:54:57Z

Can you add some unit test?

xyang16 · 2022-08-29T23:29:31Z

Can you add some unit test?

Done

frankfliu · 2022-08-30T00:12:29Z

api/src/main/java/ai/djl/util/Float16Utils.java

+     * @param order the byte order to use.
+     * @return a float array.
+     */
+    public static float[] fromByteArray(byte[] bytes, ByteOrder order) {


I'm actually not sure if this really belongs to here,

return the value between -1.0f - 1.0f seems not general enough, it might only work for specific audio data processing use case.

Maybe we should keep this in AudioUtils.java

I can move it to SampledAudioFactory

frankfliu · 2022-08-30T00:14:23Z

api/src/test/java/ai/djl/modality/audio/AudioFactoryTest.java

+
+public class AudioFactoryTest {
+
+    private static final String PATH = "src/test/resources/speech.wav";


Can we re-use test-01.wav?

Where is test-01.wav?

frankfliu · 2022-08-30T00:17:05Z

api/src/test/java/ai/djl/modality/audio/AudioFactoryTest.java

+
+    @BeforeTest
+    public void setUp() {
+        factory = AudioFactory.getInstance();


doesn't really need a setup()

frankfliu · 2022-08-30T00:18:37Z

api/src/test/java/ai/djl/modality/audio/AudioFactoryTest.java

+
+    @Test
+    public void testFromInputStream() throws IOException {
+        try (InputStream is = new BufferedInputStream(Files.newInputStream(Paths.get(PATH)))) {


Does BufferedInputStream required? If true, we need add this in our API

Yes BufferedInputStream is needed. I have added it to API.

Change-Id: Ibcd61d71c17881afc46b01abcf4f14c7b5341d15

Change-Id: Id4750f56f937cbee7e80287be8f79396aec710cf

xyang16 requested review from zachgk and frankfliu as code owners August 29, 2022 16:08

frankfliu reviewed Aug 29, 2022

View reviewed changes

xyang16 force-pushed the audio branch from d198ed7 to 00c9ebe Compare August 29, 2022 18:49

frankfliu reviewed Aug 29, 2022

View reviewed changes

xyang16 force-pushed the audio branch 4 times, most recently from 05d30f5 to cf6e419 Compare August 29, 2022 20:50

xyang16 force-pushed the audio branch 7 times, most recently from 509b309 to 63c681e Compare August 29, 2022 23:28

frankfliu reviewed Aug 30, 2022

View reviewed changes

xyang16 force-pushed the audio branch 4 times, most recently from 5c60497 to 8f110cf Compare August 30, 2022 04:10

xyang16 and others added 2 commits August 29, 2022 22:33

Add Audio API

251e65f

minor refactor

ad02622

Change-Id: Ibcd61d71c17881afc46b01abcf4f14c7b5341d15

frankfliu force-pushed the audio branch from 8f110cf to ad02622 Compare August 30, 2022 05:34

frankfliu approved these changes Aug 30, 2022

View reviewed changes

Add more unit test

4d0dea2

Change-Id: Id4750f56f937cbee7e80287be8f79396aec710cf

frankfliu force-pushed the audio branch from 2d898c2 to 4d0dea2 Compare August 30, 2022 07:02

xyang16 merged commit 6e7cafe into deepjavalibrary:master Aug 30, 2022

xyang16 deleted the audio branch May 2, 2023 19:58

	public static float[] fromByteArray(byte[] bytes, boolean bigEndian) {
	public static float[] fromByteArray(byte[] bytes, ByteOrder endian) {


		private static final Logger logger = LoggerFactory.getLogger(AudioFactory.class);

		@SuppressWarnings({"PMD.DoubleBraceInitialization", "PMD.UseConcurrentHashMap"})

	* Create new instance of audio factory from the provided factory implementation.
	* Creates new instance of audio factory from the provided factory implementation.

	* Gets {@link Audio} from file.
	* Returns {@link Audio} from file.

	* Construct a new {@code Configuration} instance.
	* Constructs a new {@code Configuration} instance.

	* Get the number of channels of an audio file.
	* Returns the number of channels of an audio file.

	* Set the number of channels for {@link AudioFactory} to use.
	* Sets the number of channels for {@link AudioFactory} to use.


		public class AudioFactoryTest {

		private static final String PATH = "src/test/resources/speech.wav";

Add Audio API #1974

Add Audio API #1974

Conversation

xyang16 commented Aug 29, 2022

Description

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 29, 2022 • edited

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xyang16 Aug 29, 2022 • edited

Choose a reason for hiding this comment

xyang16 Aug 29, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

frankfliu commented Aug 29, 2022

xyang16 commented Aug 29, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

codecov-commenter commented Aug 29, 2022 •

edited

xyang16 Aug 29, 2022 •

edited

xyang16 Aug 29, 2022 •

edited