Skip to content

Commit

Permalink
refactoring
Browse files Browse the repository at this point in the history
  • Loading branch information
chrisweb committed Oct 15, 2016
1 parent a1dc577 commit fdfa714
Show file tree
Hide file tree
Showing 14 changed files with 312 additions and 158 deletions.
5 changes: 5 additions & 0 deletions .gitattributes
@@ -0,0 +1,5 @@
# Auto detect text files and perform LF normalization
* text=auto

# JS files must always use LF for tools to work
*.js eol=lf
21 changes: 13 additions & 8 deletions .gitignore
Expand Up @@ -12,17 +12,22 @@ lib-cov
pids
results

# npm
npm-debug.log

# windows
Thumbs.db
desktop.ini

# third party libraries
node_modules
downloads
yeoman
nbproject
.sass-cache

downloads
client_build
vendor

configuration.js
# IDEs
nbproject/private
.vs
*.user
.ntvs_analysis.dat
.ntvs_analysis.dat.tmp
*.njsperf
Output-Npm.txt
63 changes: 38 additions & 25 deletions README.md
@@ -1,58 +1,71 @@
waveform-data-generator
=======================
# waveform-data-generator

Generates waveform data (peaks) that can then get visualized using: https://github.com/chrisweb/waveform-visualizer
Generates waveform data (peaks) that can then get visualized using: https://github.com/chrisweb/waveform-visualizer

![](https://github.com/chrisweb/waveform-visualizer/blob/master/examples/images/waveform.png)

Waveform created using the visualizer with data from waveform data generator
Waveform created using the visualizer with data from waveform data generator

Getting started
---------------
## Getting started

* Install git and then do a local checkout of this project
* First, start by installing nodejs (http://nodejs.org/) (which includes npm) to run the server or use the cli tool
* update npm to ensure you have the latest version installed
```npm install npm -g```
* Now install ffmeg package based on your operating system (https://www.ffmpeg.org/download.html) (and if you develop in windows add it to your path: http://www.wikihow.com/Install-FFmpeg-on-Windows)
* Do a local checkout of this project using git
* Use your command line tool and go to the root of this project (type: cd /LOCAL_PROJECT_PATH)
* type: npm install, to install the required nodejs modules (dependencies)

Launch the server
-----------------
## Launch the server

* Use your command line tool and go to the root of this project (type: cd /LOCAL_PROJECT_PATH)
* To lauch the server type: node server
* Open your browser and type the following address: 127.0.0.1:35000

Use the command line tool
-------------------------
## Using the command line tool

* Use your command line tool and go to the root of this project (type: cd /LOCAL_PROJECT_PATH)
* Type: node cli PARAMETER_1 PARAMETER_2 (...)
* For example: node cli ./downloads 1100511 ogg 200 local json false

Out of memory:
--------------
### Cli Options (Parameters)

* The first parameter "./downloads" is the repository where you want the audio files to get stored
* The second parameter "1100511" is the ID (also filename) of the track
* The third parameter "ogg" is the audio file format (also file exntension) of the track (ogg / mp3)
* The fourth parameter "200" is the amount of peaks you want to get
* The fifth parameter "local" tells the script if the file is already on your local machine, use "jamendo" to download the file from jamendo and store it the downloads directory
* The sixth parameter "json" is the type of output you want, the peaks can get outputted in either json or as a string
* The seventh parameter tells the script if it should use ffprobe to detect the track format (number of channels, the sampling frequency, ...) or use default values

## Using the web interface

Error "FATAL ERROR: JS Allocation failed - process out of memory"
* Use the following url to fetch a song from a remote source (currently only support for jamendo, fork me of you want to add another serive) and get a json containing the peaks
```http://127.0.0.1:35000/getwavedata?service=jamendo&trackId=1321406```

If the file you want to parse is too big, try increasing the memory limit of your node process, like this:
### web interface Options (Parameters)

node --max-old-space-size=1900 cli ./downloads 1100511 ogg 200 local json false
* trackId: the ID of a track [required / numeric track ID]
* trackFormat: the audio file format (file exntension) of the track (ogg or mp3) [default ogg]
* peaksAmount: the amount of peaks you want to get [default 200]
* method: the http request method to get the remote file [default GET]
* serverDirectory: the repository where you want the audio files to get stored on the server [default ./downloads]
* service: the audio file provider you want to get the track from (you can use 'local' if the file is already in your server directory) [default jamendo]
* detectFormat: tells the script if it should use ffprobe to detect the track format (number of channels, the sampling frequency, ...) or use default values (true or false) [default false]

If you still run out of memory try to reduce the sample rate by passing custom values as seventh parameter, for example if the song sample rate is 44100 but runs out of memory, then try again with 22050, like this:
Out of memory:
--------------

node --max-old-space-size=1900 cli ./downloads 1100511 ogg 200 local json sr=22050
Error "FATAL ERROR: JS Allocation failed - process out of memory"

Parameters:
If the file you want to parse is too big, try increasing the memory limit of your node process, like this:

* The first parameter "./downloads" is the repository where you want the audio files to get stored
* The second parameter "1100511" is the name of the track
* The third parameter "ogg" is the format of the track (ogg / mp3)
* The fourth parameter "200" is the amount of peaks you want to get
* The fifth parameter "local" tells the script if the file is already on your local machine, use "jamendo" to download the file from jamendo and store it the downloads directory
* The sixth parameter "json" is the type of output you want, the peaks can get outputted in either json or as a string
* The seventh parameter tells the script if it should use ffprobe to detect the track format (number of channels, the sampling frequency, ...) or use default values
node --max-old-space-size=1900 cli ./downloads 1100511 ogg 200 local json false

If you still run out of memory try to reduce the sample rate by passing custom values as seventh parameter, for example if the song sample rate is 44100 but runs out of memory, then try again with 22050, like this:

node --max-old-space-size=1900 cli ./downloads 1100511 ogg 200 local json sr=22050

TODOs
-----
Expand Down
4 changes: 0 additions & 4 deletions client/TODO

This file was deleted.

11 changes: 11 additions & 0 deletions downloads/.gitignore
@@ -0,0 +1,11 @@
*.mp3
*.ogg
*.wav
*.aiff
*.wma
*.webm
*.oga
*.opus
*.m4a
*.aac
*.flac
14 changes: 9 additions & 5 deletions package.json
Expand Up @@ -4,11 +4,12 @@
"description": "nodejs (ffmpeg) waveform data (peaks) generator",
"main": "server.js",
"scripts": {
"test": "none yet"
"test": "",
"start": "node server.js"
},
"repository": {
"type": "git",
"url": "git@github.com:chrisweb/waveform-data-generator.git"
"url": "git+ssh://git@github.com/chrisweb/waveform-data-generator.git"
},
"keywords": [
"waveform",
Expand All @@ -17,12 +18,15 @@
"data",
"generator",
"peaks",
"ffmpeg"
"ffmpeg"
],
"author": "Chris Weber (chrisweb) <chris963@gmail.com>",
"author": "Chris Weber <chris963@gmail.com> (chrisweb)",
"license": "MIT",
"bugs": {
"url": "https://github.com/chrisweb/waveform-data-generator/issues"
},
"homepage": "https://github.com/chrisweb/waveform-data-generator"
"homepage": "https://github.com/chrisweb/waveform-data-generator",
"dependencies": {
"marked": "0.3.5"
}
}
68 changes: 47 additions & 21 deletions server/bootstrap.js
Expand Up @@ -8,8 +8,10 @@ var querystring = require('querystring');

var waveformData = require('./library/waveformData');

var serverPort = 35000;
var serverIp = '127.0.0.1';
var marked = require('marked');

var serverPort = process.env.PORT || 35000;
var serverIp = process.env.HOSTNAME || '127.0.0.1';

/**
*
Expand All @@ -18,20 +20,18 @@ var serverIp = '127.0.0.1';
* @param {type} request
* @param {type} response
*/
http.createServer(function (request, response) {
var server = http.createServer(function (request, response) {

// parse the url
var urlParts = url.parse(request.url);

//console.log(urlParts);

// check if its is the url of a javascript file
if (urlParts.pathname.split('.').pop() === 'js') {

// if the file exists send it to the client
// not really secure but this is a prototype
// TODO: filter the file request
fs.readFile('client' + urlParts.pathname, function(error, fileContent) {
fs.readFile('client' + urlParts.pathname, function (error, fileContent) {

if (!error) {

Expand All @@ -52,14 +52,14 @@ http.createServer(function (request, response) {
});

} else {

// handle the "routes"
switch(urlParts.pathname) {
switch (urlParts.pathname) {
case '/':
fs.readFile('client/index.html', function(error, html) {
/*fs.readFile('client/index.html', function (error, html) {
if (!error) {

// send the main html page to the client
response.writeHead(200, { 'Content-Type': 'text/html' });
response.write(html);
Expand All @@ -75,15 +75,37 @@ http.createServer(function (request, response) {
}
});*/

fs.readFile('README.md', 'utf-8', function (error, document) {

if (!error) {

// send the main html page to the client
response.writeHead(200, { 'Content-Type': 'text/html' });
response.write(marked(document));
response.end();

} else {

// the main page could not be found return a page not
// found message
response.writeHead(404, { 'Content-Type': 'text/html' });
response.write('page not found');
response.end();

}

});

break;
case '/getwavedata':

var queryObject = querystring.parse(urlParts.query);

if (typeof queryObject !== 'undefined') {

waveformData.getRemoteWaveData(queryObject, function(error, peaks) {
waveformData.getRemoteWaveData(queryObject, function (error, peaks) {

if (!error) {

Expand Down Expand Up @@ -113,32 +135,32 @@ http.createServer(function (request, response) {
//console.log(queryObject);

if (typeof queryObject !== 'undefined' && queryObject.trackId !== 'undefined' && queryObject.trackFormat !== 'undefined') {

var trackName = queryObject.trackId + '.' + queryObject.trackFormat;

if (queryObject.serverDirectory === undefined) {

queryObject.serverDirectory = './downloads';

}

var trackPath = queryObject.serverDirectory + '/' + trackName;

var fileStat = fs.statSync(trackPath);

var mimeType;

switch(queryObject.trackFormat) {
switch (queryObject.trackFormat) {
case 'ogg':
mimeType = 'audio/ogg';
break;
case 'mp3':
mimeType = 'audio/mpeg';
break;
}

response.writeHead(200, { 'Content-Type': mimeType, 'Content-Length': fileStat.size });

var readStream = fs.createReadStream(trackPath);

readStream.pipe(response);
Expand All @@ -154,6 +176,10 @@ http.createServer(function (request, response) {
}
}

}).listen(serverPort, serverIp);
});

server.listen(serverPort, serverIp, function () {

console.log('server is listening, ip: ' + serverIp + ', port: ' + serverPort);

console.log('server is listening, ip: ' + serverIp + ', port: ' + serverPort);
});

0 comments on commit fdfa714

Please sign in to comment.