Investigate existing Extendables framework test suite pass-ability

[daluu]

Unless I'm executing the tests incorrectly...(as I'm just running the runner.someName.jsx scripts under test directory of Extendables via ExtendScript toolkit)...

It appears at last check, the test framework's included test suite doesn't fully pass, under Adobe suite (executed from ExtendScript toolkit) CS6. The failed test cases are noted below, with some additional info.

Would be nice to look into whether the suite ever 100% passed before, and under what Adobe versions does the test get failures, etc. Would be nice to get community input on this as I don't have all the supported/supportable Adobe versions to test with.

Test framework failures:

is entirely CommonJS-compliant
Expected 'index' to equal 'package-b'

has a working code sample that does XML traversal
SyntaxError: Expected: ) in ~/Documents/GitHub/Extendables/examples/xml-traversal.jsx (line 3)

OS: Windows 7/64 6.1 Service Pack 1 (32-bit emulation)
ExtendScript build: 66.489054
ExtendScript version: 4.2.12
path: ~
locale: en_US
app: ExtendScript Toolkit
app version: 3.8

Package test failures:

Template package
has unit tests
Expected false to be truthy.

HTTP requests
has a high-level shortcut for GET requests
Expected 404 to equal 200.
has a high-level shortcut for HEAD requests
Expected 404 to equal 200.
has a high-level shortcut for POST requests
Expected 404 to equal 200.
has a high-level shortcut for DEL requests
Expected 404 to equal 200.
can follow redirects
Expected 404 to equal 200.
can follow redirects manually, using the .follow() method
Expected 404 to equal 200.
is thoroughly stress-tested
Expected 404 to equal 200.

Patch test failures:

runner.patches.jsx fails to complete test execution with the following exception in ExtendScript toolkit:

"result.toLowerCase is not a function" for the following line of code:
line 130 of object.conversions.jsx: 'lower': function () { return result.toLowerCase(); },

[daluu]

Also tried executing tests of the framework runner via an actual Adobe app rather than ExtendScript toolkit, same issue. And worse perhaps as some apps don't support sockets to test the rest of the framework tests, like Adobe Illustrator and Adobe Photoshop.

So in those cases, only able to execute main framework tests. Debug/app info:

OS: Windows 7/64 6.1 Service Pack 1
ExtendScript build: 66.490082
ExtendScript version: 4.2.12
path:
locale: en_US
app: Adobe Illustrator
app version: 16.0.0

[daluu]

On a related note, I have Adobe InDesign CS6 (32-bit on 64-bit PC) installed as part of suite on work PC, but either not available or need to figure out how to run script from it. Just drag & drop scripts into the app doesn't seem to work unlike Adobe Illustrator & Photoshop. For InDesign I get error:

Cannot open the file "runner.framework.jsx". Adobe InDesign may not support the file format, a plug-in that supports the file format may be missing, or the file may be open in another application.

the script isn't open in another app (also checked the process isn't still stuck running in memory), so strange, no idea of problem.

[daluu]

I can attempt to fix the failing tests as well, but I'll generally patch broken things that affect me as a priority first.

Although one reason I filed this, is that making changes to the core framework, the existing test suite is useful to use as a benchmark against whether the changes/refactoring has broken the framework or not, and the current failures (before any changes) impact that a bit.

[daluu]

According to Stijn Debrouwere, the tests did pass before for InDesign CS3. I don't have InDesign CS3 to check against though.

[daluu]

Seems tests don't run on Photoshop CS2 or the ExtendScript Toolkit for that (toolkit version 1.0.0 36.182408.61, ExtendScript version 3.6.51).

In Photoshop get error:

Error 48: File or folder does not exist.
Line: 0
-> // we need to log somethings before the log module is loaded,

in the toolkit, it fails to run saying file or folder does not exist for the relative path to extendables.jsx. Making it absolute path didn't seem to help.

[daluu]

Testing on CS6 on Mac, patch tests run and complete, here are the results in terms of failures seen:

Suite: Patches: application (1 tests, 1 failures)
can determine which host app is running
ReferenceError: result.toLowerCase is not a function in ~/Documents/Extendables-master/patches/object.conversions.jsx (line 130)

Suite: Patches: object serialization (4 tests, 2 failures)
can base64-encode utf8 data
Expected 'tested' to equal false.
can base64-encode binary data
Expected 'tested' to equal false.

[daluu]

Results for

OS: Macintosh OS 10.7.5/64
ExtendScript build: 61.423205
ExtendScript version: 4.1.23
locale: en_US
app: Adobe Photoshop
app version: 12.0.0

and

OS: Macintosh OS 10.7.5
ExtendScript build: 61.422079
ExtendScript version: 4.1.23
locale: en_US
app: ExtendScript Toolkit
app version: 3.5

Same failures as ESTK v3.8 but patch tests don't get stuck or fail with exception. Results have failures though:

Suite: Patches: application (1 tests, 1 failures)
FAILED can determine which host app is running
ReferenceError: result.toLowerCase is not a function in ~/Downloads/Extendables-master/patches/object.conversions.jsx (line 130)

Suite: Patches: object serialization (4 tests, 2 failures)
FAILED can base64-encode utf8 data
Expected 'tested' to equal false.
FAILED can base64-encode binary data
Expected 'tested' to equal false.

Also came across this error once trying to run patch tests in Photoshop:

Error 2: tests is undefined.
Line: 14
-> tests.to_html("tests.patches.html");