TIMOB-9379 initial push of Anvil driver for all platforms #2323

rusticphilosopher · 2012-06-05T16:08:53Z

Test instructions for this are limited to running the driver against all 3 platforms to ensure that the test suites are executed properly.

Once node.js is installed, inside the support/driver directory run:

"node driver.js --mode=local --platform=android"
once running and a android device is plugged in, input "start"

Repeat the above steps for both iOS and Mobile Web. For iOS, the simulator will be started.

** For mobile web, an Android device must be connected and seen by ADB but also the device must have access to the wifi network and be able to see the node.js web server being run by the driver.

Whoever does the functional test for MW might just want to talk to me regarding the setup.

rusticphilosopher · 2012-06-05T17:06:06Z

hold merging for further updates

robsjc · 2012-06-12T21:07:20Z

Hi Opie,

Attached are scripts called from Jenkins Master box.

Driver-LP.sh – is core Master script – this is simply called from Jenkins Application with only one param: $GIT_BRANCH.

The 'runtest---' scripts run on the slave box as Jenkins slave modules. They are called directly from Jenkins too. NOT as part of the Modules build (as mentioned in previous email) but it is means for offloading any Drillbit application performance hits. The slave module for Drillbit also requires two additional scripts not included here. They are for pre-config and post-processing.

Robert Collazo

QA Automation Engineer

Appcelerator, Inc.

rcollazo@appcelerator.commailto:rcollazo@appcelerator.com

unknown.png

The Next Generation Mobile Platform

http://www.appcelerator.comhttp://www.appcelerator.com/
#!/bin/sh

PLATFORM=ANDROID
PLATFORM_CLASS=unknown
VERSION=unknown

DATA_TARGET=../testdata-$PLATFORM
VERSION_DIR=cat slave_version.txt

if [ ! -e $DATA_TARGET ]
then
mkdir $DATA_TARGET
fi

====== ANDROID Start Up section =============================================

killall logcat

killall emulator

start logcat - as backgroud process

/Users/rcollazo/Downloads/android-sdk-mac_x86/platform-tools/adb -s emulator-5560 logcat &

start emuator - need to exec as forground process (???)

/Users/rcollazo/Downloads/android-sdk-mac_x86/tools/emulator-arm -avd titanium_4_HVGA -port 5560 -sdcard /Users/rcollazo/.titanium/titanium_4_HVGA.sdcard -logcat :d,,TiAPI:V -no-boot-anim -partition-size 128

wait_for_connection # ./ monitor_adb.py

NOTE: this is temp Hack for Android - file bug to package icon -OR- trap File not Found Error

cp -p $HOME/appicon.png /Users/rcollazo/build/workspace/slave_job2/$VERSION_DIR/titanium_mobile/build/drillbit/Drillbit.app/Contents/Resources/test_harness/build/android/bin/assets/Resources/appicon.png

=============================================================================

./drillbit/drillbit.py --platforms=android --tests=analytics,android.filesystem,android.geolocation,android.kroll,android.network.httpclient,android.notificationmgr,android.platform,android.resources,android.string,codec,database,json,kroll,locale,network,network.httpclient,network.socket,network.socket.tcp,platform,properties,titanium,ui.2dmatrix,ui.controls,yahoo --reset-config --android-runtime=v8 --android-device=emulator-5560 --autorun --results-dir=../testdata-$PLATFORM

zip -rq testdata.zip $DATA_TARGET
pwd
ls -la ../..
mv testdata.zip ../..

#!/bin/sh

PLATFORM=IOS
PLATFORM_CLASS=iphone
VERSION=5.0

DATA_TARGET=../testdata-$PLATFORM

if [ ! -e $DATA_TARGET ]
then
mkdir $DATA_TARGET
fi

for TEST_NAME in analytics
app
blob
buffer
codec
coonsole
database
facebook
filesystem
includes
json
jss
kroll
locale
media
network
network.httpclient
network.socket
network.socket.tcp
platform
properties
stream
titanium
ui
ui.2dmatrix
ui.controls
ui.layout
utils
xml
yahoo
do
./drillbit/drillbit.py --platforms=$PLATFORM_CLASS --tests=$TEST_NAME --results-dir=$DATA_TARGET --$PLATFORM-version=$VERSION --autorun --autoclose
done

zip -rq testdata.zip $DATA_TARGET
pwd
ls -la ../..
mv testdata.zip ../..

#!/bin/sh

A hudson Build driver for Titanium Mobile

this is pacakge for Drillbit Testing

SLAVE_PACKAGE='slave_package.zip'

export PATH=/bin:/usr/bin:$PATH

$1 from Jenkins Application

GIT_BRANCH=$1

Naming conventions...

TARGET_BRANCH=titanium_mobile_$GIT_BRANCH
GIT_REVISION=git log --pretty=oneline -n 1 | sed 's/ .*//' | tr -d '\n' | tr -d '\r'
VERSION=python $TITANIUM_BUILD/common/get_version.py | tr -d '\r'
TIMESTAMP=date +'%Y%m%d%H%M%S'
VTAG=$VERSION.v$TIMESTAMP
BASENAME=dist/mobilesdk-$VTAG

Do Core Build

scons package_all=1 version_tag=$VTAG $TI_MOBILE_SCONS_ARGS

if [ "$PYTHON" = "" ]; then
PYTHON=python
fi

Naming conventions...

SDK_ARCHIVE="$BASENAME-osx.zip"

Name mapping for DIR Paths vs Git Branch Names

TARGET_EXT='master'

$GIT_BRANCH_MASTER = "master" = $TARGET_EXT

if [ $GIT_BRANCH = '2_0_X' ]
then
TARGET_EXT='2.0.x'
fi

if [ $GIT_BRANCH = '1_8_X' ]
then
TARGET_EXT='1.8.x'
fi

if [ $GIT_BRANCH = '1_7_X' ]
then
TARGET_EXT='1.7.x'
fi

cd $TITANIUM_BUILD/jobs/titanium_mobile_$TARGET_EXT/workspace

verify scons created Build

if [ -e "$SDK_ARCHIVE" ]
then
echo "$SDK_ARCHIVE - does Exist."
else
echo "Missing SDK zip file... Exiting."
exit 1
fi

kludge for preserving version number - re-used when unpackaged on slave

echo $VTAG > slave_version.txt

if [ -e $SLAVE_PACKAGE ]
then
echo "removing previous package file."
rm -r $SLAVE_PACKAGE
fi

build minimal package for Drillbit drop

zip -rq $SLAVE_PACKAGE slave_version.txt build drillbit

if [ -e "tmp_unbundle" ]
then
echo "removing old tmp files."
rm -r tmp_unbundle
fi

Unzip Build

mkdir tmp_unbundle
mkdir tmp_unbundle/dist
cd tmp_unbundle/dist
unzip -o ../../$SDK_ARCHIVE

Extract additional minimals for Drillbit Package

cd mobilesdk/osx
mv $VTAG $VERSION
cd ../../..

append to Drillbit Package

zip -urq ../$SLAVE_PACKAGE dist/mobilesdk

copy and timestamp the Drillbit Package for short term archive

cd $TITANIUM_BUILD/jobs/titanium_mobile_$TARGET_EXT/workspace
TS=date +"%m%d%y-%H%M%S"
ARCHIVE_FNAME="slave_package-$TS.zip"
cp $SLAVE_PACKAGE $ARCHIVE_FNAME

push core Build packages to Amazon S3 buckets

$PYTHON $TITANIUM_BUILD/common/s3_cleaner.py mobile $GIT_BRANCH
$PYTHON $TITANIUM_BUILD/common/s3_uploader.py mobile $BASENAME-osx.zip $GIT_BRANCH $GIT_REVISION $BUILD_URL
$PYTHON $TITANIUM_BUILD/common/s3_uploader.py mobile $BASENAME-linux.zip $GIT_BRANCH $GIT_REVISION $BUILD_URL
$PYTHON $TITANIUM_BUILD/common/s3_uploader.py mobile $BASENAME-win32.zip $GIT_BRANCH $GIT_REVISION $BUILD_URL

… web support, initial push or test suites and numerous other architecture changes

sptramer · 2012-06-18T19:46:54Z

Will CR + iOS FR.

sptramer · 2012-06-18T19:52:19Z

driver/README

+added to the "harnessTemplate/Resources/suites" directory.  An example/template for a suite exists at the top
+level driver directory.
+
+In order to add new tests to a suite, the test should be added to the "tests" array within the 


Not necessary for the initial rev, but it would be good to autogenerate this (or dynamically determine at runtime) in the future.

given the static nature of a suite file I didn't think it was strong enough reason to justify getting into code generation. I am open to further discussion on this though.

This is one of those things that might need to change in a later rev if we add user-defined tests for their own harness/app for testing their individual components.

Yeah, I will feel better crossing that bridge when we get closer to that and have some more concrete user stories / full picture. Personally I would rather avoid over designing for future behavior we don't have a full picture of yet. We might address a change here but I think we should leave this be for the time being since it's simple, obvious, extremely low overhead and works. My 2 cents.

sptramer · 2012-06-18T21:35:50Z

Going to hold further review at @DizzyMonkey's request

… update to test suites in order to support stronger test specific scopes

sptramer · 2012-06-20T17:46:14Z

driver/common.js

+	this.createHarness = function(platform, command, successCallback, errorCallback) {
+		var createCallback = function() {
+			try {
+				fs.mkdirSync(driverGlobal.tempDir + "/harness/" + platform, 0777);


Should we be explicitly require()ing node.js components which may be loaded by other commonjs modules? Seems like good practice.

I tried this previously (and just verified again) and the required module cannot be seen inside other required modules

That is incredibly stupid. Did you file a bug with node?

sptramer · 2012-06-21T21:50:50Z

driver/example.cfg

+	refer to README file for more info on configurations
+	NOTE: please refer to node.js documentation for info on "__dirname"
+	*/
+	this.harnessConfigsDir = ""; // location of the harness configurations;


This can probably safely point to the example above, since most people will be running the included test suite with the provided configs.

Good catch. Yeah, that is accidental as the configs dir really shouldn't be a item in the cfg file since that is part of the checked in test content. This might change in the future but for now...

I actually removed several items from the config file in order to follow a "less rope" mindset and have the config file put more focus on properties that actually need attention. I also included very minor updates to the README to support some of the changes to the example config file.

billdawson · 2012-06-21T22:00:20Z

Done with initial review, will wait for more updates. General comments:

There are a lot of instances of violations of our JavaScript standard (though I don't think there are a lot of different types of violations.) All the if( and for( (lack of space) are violations. And the lack of semi-colons as well at the end of var x = function(){...}; statements. (Those are the only two types of violations I can think of.)
There seems to be a mix of usage of !== instead of != (and === and ==). It should be consistent, and probably should almost always be the strict (in)equality operator.
The code comments are great and much better than what we usually do. But I would like the extra polish of consistent proper case (sentences beginning with upper-case letters.)

sptramer · 2012-06-21T22:06:11Z

driver/driver.js

+
+	var platform = util.getArgument(process.argv, "--platform");
+	try {
+		driverGlobal.platform = require(__dirname + "/platforms/" + platform);


Need a sanity check here in case the platform isn't provided, (is undefined) so that usage can be displayed.

usage is already displayed if the platform argument is not displayed or if the platform is unrecognized (fails to load module). The error though is a general fail to load error and I have updated to provide more focused logging and error handling.

sptramer · 2012-06-21T22:15:34Z

iOS functional test; the tests ran with the default configuration (124 pass 19 fail) but there is an issue. When ios-sim crashes/aborts because it can't start a simulator in time, anvil seems to not respond. The message is

[DEBUG] Session could not be started: Error Domain=DTiPhoneSimulatorErrorDomain Code=2 "Simulator session timed out." UserInfo=0x104002dc0 {NSLocalizedDescription=Simulator session timed out.}

Although this is hard to reproduce. It may be the case that we need to return a specific error code from ios-sim and anvil needs to look for it.

cb1kenobi · 2012-06-21T23:12:47Z

driver/driver.js

+		process.exit(1);
+	}
+
+	var configString = fs.readFileSync(configFilePath);


We should just have the config file as a module that we can require(). That way we don't have to do all this sync file reading and evaling.

Upon further thought, I went ahead and changed the config to a required module.

cb1kenobi · 2012-06-21T23:19:35Z

driver/driver.cfg (or whatever it's going to be called) should be added to the .gitignore file.

…nization and fix porting issues

… the json test suite

sptramer · 2012-06-22T17:12:35Z

Got the following error during a functional test:

git@[pull-2323:041d30a] ⤅ ∃? % node driver.js --platform=ios
exception <TypeError: Bad argument> occurred when creating undefined
exception <Error: ENOENT, no such file or directory 'undefined/harness'> occurred when creating undefined/harness
exception <Error: ENOENT, no such file or directory 'undefined/logs'> occurred when creating undefined/logs
exception <Error: ENOENT, no such file or directory 'undefined/logs/ios'> occurred when creating undefined/logs/ios

This was from an old-style config migrated to a config.js which indicates that we might want some error checking for values from the config to make sure they're not undefined.

sptramer · 2012-06-22T17:23:19Z

Unable to reproduce the ios-sim error and the logs for it were helpfully nuked by an os x reboot (which cleans most of /tmp). Probably just going to have to be an exit status check. Any bad exit returns the EXIT_FAILURE (system-dependent value).

billdawson · 2012-06-22T18:02:17Z

Maybe we can fit this in before merge:

An error message if no device/emulator is available when start is called. I called start and walked away for a bit, and came back and it was still just sitting there not doing anything. (Then I realized my sorry a$$ hadn't started an emulator or plugged in a device.)

Future "nice to haves":

For whatever reason I didn't notice the passing around of testRun (in tests) until today. I think it's pretty awkward and mysterious. If there's anything we can do to get rid of that (and have everything still work, of course), it would be a win in my book.
Strict equality consistency (===, !==)
Config.js setting for default platform. Just for kicks and laughs. Like I'll almost always be running android.

sptramer · 2012-06-22T18:13:20Z

Functional tested ios and code reviewed; a few more comments.

sptramer · 2012-06-22T18:21:57Z

Two more high-profile failures for ios/os x:

Log files are always empty (/var/driver/logs/ios)
When the --sim-version argument is specified for a run, it only affects the first configuration. Subsequent configurations use the default.

billdawson · 2012-06-22T21:14:37Z

My log files are also empty, btw.

sptramer · 2012-06-22T21:27:11Z

After the latest update, ios is no longer reporting results:

ios-sim closed
iPhone simulator closed

RESULTS SUMMARY:
    Results count:
        PASSED - 0
        FAILED - 0

sptramer · 2012-06-22T21:45:03Z

Functional tested & code reviewed for ios. APPROVED.

billdawson · 2012-06-22T22:06:32Z

cr/fr accepted

cb1kenobi · 2012-06-22T23:05:30Z

Code reviewed and tested. Accepted.

TIMOB-9379 initial push of Anvil driver for all platforms

TIMOB-9379 initial push of Anvil driver for all platforms

8311860

Opie Cyrus added 3 commits June 18, 2012 00:00

TIMOB-9379 major update to Driver to support logging, revamped mobile…

5e55b6d

… web support, initial push or test suites and numerous other architecture changes

Merge remote-tracking branch 'appcelerator/master' into timob-9379

031359c

TIMOB-9379 update to address timing issue

f35e82e

sptramer reviewed Jun 18, 2012
View reviewed changes

Opie Cyrus added 3 commits June 19, 2012 02:59

TIMOB-9379 update of timing behavior to address edge cases along with…

1de56cb

… update to test suites in order to support stronger test specific scopes

Merge remote-tracking branch 'appcelerator/master' into timob-9379

cf7b258

Merge remote-tracking branch 'appcelerator/master' into timob-9379

3085bb4

sptramer reviewed Jun 20, 2012
View reviewed changes

sptramer reviewed Jun 21, 2012
View reviewed changes

cb1kenobi reviewed Jun 21, 2012
View reviewed changes

Opie Cyrus added 4 commits June 22, 2012 01:14

TIMOB-9379 multiple changes in response to PR feedback

8b13738

TIMOB-9379 a few changes to the test suites in order to clean up orga…

adcadff

…nization and fix porting issues

Merge remote-tracking branch 'appcelerator/master' into timob-9379

8720a59

TIMOB-9379 fixed porting syntax issue that caused a failure on MW for…

1ed50a1

… the json test suite

TIMOB-9379 more updates based on feedback

a674d20

TIMOB-9379 fix for result output

f9be166

TIMOB-9379 change mw to mobileweb

d008cdf

cb1kenobi added a commit that referenced this pull request Jun 22, 2012

Merge pull request #2323 from dizzymonkey/timob-9379

3432004

TIMOB-9379 initial push of Anvil driver for all platforms

cb1kenobi merged commit 3432004 into tidev:master Jun 22, 2012

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TIMOB-9379 initial push of Anvil driver for all platforms #2323

TIMOB-9379 initial push of Anvil driver for all platforms #2323

rusticphilosopher commented Jun 5, 2012

rusticphilosopher commented Jun 5, 2012

robsjc commented Jun 12, 2012

sptramer commented Jun 18, 2012

sptramer Jun 18, 2012

rusticphilosopher Jun 20, 2012

sptramer Jun 20, 2012

rusticphilosopher Jun 21, 2012

sptramer commented Jun 18, 2012

sptramer Jun 20, 2012

rusticphilosopher Jun 20, 2012

sptramer Jun 20, 2012

sptramer Jun 21, 2012

rusticphilosopher Jun 22, 2012

rusticphilosopher Jun 22, 2012

billdawson commented Jun 21, 2012

sptramer Jun 21, 2012

rusticphilosopher Jun 22, 2012

sptramer commented Jun 21, 2012

cb1kenobi Jun 21, 2012

rusticphilosopher Jun 22, 2012

cb1kenobi commented Jun 21, 2012

sptramer commented Jun 22, 2012

sptramer commented Jun 22, 2012

billdawson commented Jun 22, 2012

sptramer commented Jun 22, 2012

sptramer commented Jun 22, 2012

billdawson commented Jun 22, 2012

sptramer commented Jun 22, 2012

sptramer commented Jun 22, 2012

billdawson commented Jun 22, 2012

cb1kenobi commented Jun 22, 2012

TIMOB-9379 initial push of Anvil driver for all platforms #2323

TIMOB-9379 initial push of Anvil driver for all platforms #2323

Conversation

rusticphilosopher commented Jun 5, 2012

rusticphilosopher commented Jun 5, 2012

robsjc commented Jun 12, 2012

====== ANDROID Start Up section =============================================

killall logcat

killall emulator

start logcat - as backgroud process

/Users/rcollazo/Downloads/android-sdk-mac_x86/platform-tools/adb -s emulator-5560 logcat &

start emuator - need to exec as forground process (???)

/Users/rcollazo/Downloads/android-sdk-mac_x86/tools/emulator-arm -avd titanium_4_HVGA -port 5560 -sdcard /Users/rcollazo/.titanium/titanium_4_HVGA.sdcard -logcat :d,,TiAPI:V -no-boot-anim -partition-size 128

wait_for_connection # ./ monitor_adb.py

NOTE: this is temp Hack for Android - file bug to package icon -OR- trap File not Found Error

=============================================================================

A hudson Build driver for Titanium Mobile

this is pacakge for Drillbit Testing

$1 from Jenkins Application

Naming conventions...

Do Core Build

Naming conventions...

Name mapping for DIR Paths vs Git Branch Names

$GIT_BRANCH_MASTER = "master" = $TARGET_EXT

verify scons created Build

kludge for preserving version number - re-used when unpackaged on slave

build minimal package for Drillbit drop

Unzip Build

Extract additional minimals for Drillbit Package

append to Drillbit Package

copy and timestamp the Drillbit Package for short term archive

push core Build packages to Amazon S3 buckets

sptramer commented Jun 18, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sptramer commented Jun 18, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

billdawson commented Jun 21, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sptramer commented Jun 21, 2012

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cb1kenobi commented Jun 21, 2012

sptramer commented Jun 22, 2012

sptramer commented Jun 22, 2012

billdawson commented Jun 22, 2012

sptramer commented Jun 22, 2012

sptramer commented Jun 22, 2012

billdawson commented Jun 22, 2012

sptramer commented Jun 22, 2012

sptramer commented Jun 22, 2012

billdawson commented Jun 22, 2012

cb1kenobi commented Jun 22, 2012