Failing test that prun does not clobber string escapes

[jstenar]

Running the master branch 0ceba04 I get a failed test (see below). This is using python 2.6 (32bit) on windows 7 64-bit.

FAIL: Test that prun does not clobber string escapes (GH #1302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "c:\python26\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "c:\python\external\ipython\IPython\testing\decorators.py", line 228, in skipper_func
    return f(*args, **kwargs)
  File "c:\python\external\ipython\IPython\core\tests\test_magic.py", line 396, in test_prun_quotes
    nt.assert_equal(_ip.user_ns['x'], '\t')
AssertionError: ' ' != '\t'
>>  raise self.failureException, \
      (None or '%r != %r' % (' ', '\t'))

[fperez]

For the record, I don't see it on linux...

[takluyver]

This is magic argument parsing again. I think after 0.13 we'll need to think up a better solution to this whole thing.

@jstenar : You may have seen recently that @Carreau set up a Mac to run the test suite overnight (discussed on #1328). If you've got a Windows machine running overnight that has some spare time, it would be great to have this for Windows as well. It just needs a script that will pull from github, install into a virtualenv, run iptest with some options, zip up the result files and put them on the web where they can easily be downloaded.

[jstenar]

Thomas skrev 2012-02-22 14:30:

This is magic argument parsing again. I think after 0.13 we'll need to think up a better solution to this whole thing.

@jstenar : You may have seen recently that @Carreau set up a Mac to run the test suite overnight (discussed on #1328). If you've got a Windows machine running overnight that has some spare time, it would be great to have this for Windows as well. It just needs a script that will pull from github, install into a virtualenv, run iptest with some options, zip up the result files and put them on the web where they can easily be downloaded.

I don't have a machine that is always running but I could have a script
that runs those evenings I have the machine running.

Do you have a suggestion for where I can upload it? Is it not possible
to upload directly to shiningpanda?

/Jörgen

[takluyver]

Annoyingly, although Jenkins allows triggering a build from a script, and it allows uploading a file for a build, it doesn't allow both together: you have to upload the file manually via a web form.

Maybe @fperez would let us use a bit of space on the server that hosts the wiki and the archive. Fernando, the zip file is a bit over 100KB at present, and it would be uploaded and downloaded once per day. Hopefully there's an easy way to do something like scp from Windows.

[fperez]

Of course! The hosting plan I use is 'unlimited' both in bandwidth and storage, so unless we're talking about terabytes, you don't even have to ask :) Just go for it. @jstenar, if you need to be the one doing the uploading, email me your public SSH key and I'll add it to the server.

[jstenar]

Fernando Perez skrev 2012-02-22 20:40:

Of course! The hosting plan I use is 'unlimited' both in bandwidth and storage, so unless we're talking about terabytes, you don't even have to ask :) Just go for it. @jstenar, if you need to be the one doing the uploading, email me your public SSH key and I'll add it to the server.

I can log in to the machine now. Where do you want me to put the testlogs?

I will try to set up a testscript over the weekend.

/Jörgen

[fperez]

I'll let @takluyver answer that, as he's the one who will be handling their integration with Shining Panda. Thanks to everyone for this!

[jstenar]

Thomas skrev 2012-02-22 14:30:

This is magic argument parsing again. I think after 0.13 we'll need to think up a better solution to this whole thing.

@jstenar : You may have seen recently that @Carreau set up a Mac to run the test suite overnight (discussed on #1328). If you've got a Windows machine running overnight that has some spare time, it would be great to have this for Windows as well. It just needs a script that will pull from github, install into a virtualenv, run iptest with some options, zip up the result files and put them on the web where they can easily be downloaded.

---

Reply to this email directly or view it on GitHub:
#1428 (comment)

@Carreau could you share the script you use to run the testsuite. I
won't be able to use it as is but at least it will show me one way of
doing this.

@thomas when using a virutalenv what is the best way to install ipython
so it can be unistalled cleanly?

/Jörgen

[takluyver]

The script that we use on ShiningPanda is below (I sent @Carreau this, so I think he adapted it for Mac). It runs with the virtualenv active, and a fresh checkout of github master as the working directory.

If you just install IPython inside the virtualenv (python setup.py install), it shouldn't interfere with your normal installation at all, so it's safe to leave it. If you need to remove it, you can delete the folder the virtualenv is based in, or call pip uninstall ipython with the virtualenv active.

Upload location: if you can put it somewhere on archive.ipython.org - maybe make a new testresults folder, and call it testresults-win-latest.zip. I'll set ShiningPanda up so that you make an HTTP request when you've uploaded the file, which will trigger it to download and add the results.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/slave/local/lib
export ZMQ_DIR=/home/slave/local
pip install nose coverage
pip install pyzmq
pip install readline

rm -rf build/
python setup.py install

mkdir -p results
cd results
rm -f *.xunit.xml
rm -f *.coverage.xml
iptest --with-xml-coverage --with-xunit

This will create *.xunit.xml and *.coverage.xml files, which you can zip and upload.

[Carreau]

Le 23 fvr. 2012 21:55, Jrgen Stenarson a crit :

Thomas skrev 2012-02-22 14:30:

This is magic argument parsing again. I think after 0.13 we'll need to think up a better solution to this whole thing.

@jstenar : You may have seen recently that @Carreau set up a Mac to run the test suite overnight (discussed on #1328). If you've got a Windows machine running overnight that has some spare time, it would be great to have this for Windows as well. It just needs a script that will pull from github, install into a virtualenv, run iptest with some options, zip up the result files and put them on the web where they can easily be downloaded.

---

Reply to this email directly or view it on GitHub:
#1428 (comment)

@Carreau could you share the script you use to run the testsuite. I
won't be able to use it as is but at least it will show me one way of
doing this.

Sure, put them as a attachement.

You can surely improve them.
weekly-ipython totally remove the virtualenv and rebuild it.
daily-ipython try to clean as few files as possible to limit bandwidth usage and time reinstalling all python package.
the only catch I had was that cron need a passwordless sshkey to login.

As a side note, the build failed for an unknown reason this morning (xml files where not created.) but is working again so don't take today's Shining Panda test into account.

Matthias

@thomas when using a virutalenv what is the best way to install ipython
so it can be unistalled cleanly?

/Jrgen

---

Reply to this email directly or view it on GitHub:
#1428 (comment)

[takluyver]

Let us know if that failure happens again, @Carreau , and we'll see if we can work it out.

[jstenar]

Thomas skrev 2012-02-24 00:12:

iptest --with-xml-coverage --with-xunit

This will create *.xunit.xml and *.coverage.xml files, which you can zip and upload.

I have trouble getting the coverage to work. All tests fail.
I have coverage 3.5.1 installed

(Py26) C:\python\ipydevel\VENV\ipython\results> iptest --with-xml-coverage

---

IPython test group: IPython.config
no such option: -u
Use 'coverage help' for help.
No data to report.
.
.
SNIP
.
.

---

Test suite completed for system with the following information:
{'commit_hash': 'cea9772',
'commit_source': 'installation',
'ipython_path':
'C:\python\ipydevel\VENV\Py26\lib\site-packages\ipython-0.13.dev-py2.6.egg\IPython',
'ipython_version': '0.13.dev',
'os_name': 'nt',
'platform': 'Windows-7-6.1.7601-SP1',
'sys_executable': 'C:\python\ipydevel\VENV\Py26\Scripts\python.exe',
'sys_platform': 'win32',
'sys_version': '2.6.6 (r266:84297, Aug 24 2010, 18:46:32) [MSC v.1500
32 bit (Intel)]'}

Tools and libraries available at test time:
sqlite3 zmq

Tools and libraries NOT available at test time:
curses matplotlib pexpect pymongo qt tornado wx wx.aui

Ran 10 test groups in 2.581s

Status:

ERROR - 10 out of 10 test groups failed.

Runner failed: IPython.config
You may wish to rerun this one individually, with:
coverage run --source=IPython.config -u
C:\python\ipydevel\VENV\Py26\lib\site-packages\ipython-0.13.dev-py2.6.egg\IPython\testing\iptest.py
IPython.config
.
.
.

[takluyver]

It looks like the -u is a Python flag, introduced in IPython.utils.process, line 103. That talks about twisted, so maybe it's no longer necessary - can you try taking it out and rerunning the tests?

[jstenar]

Thomas skrev 2012-02-24 21:24:

It looks like the -u is a Python flag, introduced in IPython.utils.process, line 103. That talks about twisted, so maybe it's no longer necessary - can you try taking it out and rerunning the tests?

Now it works!

[jstenar]

Thomas skrev 2012-02-24 21:24:

It looks like the -u is a Python flag, introduced in IPython.utils.process, line 103. That talks about twisted, so maybe it's no longer necessary - can you try taking it out and rerunning the tests?

---

Reply to this email directly or view it on GitHub:
#1428 (comment)

I tried running the tests for python3.2 but it seems a bunch of examples
are not converted by 2to3. I see several errors like the one below
during python setup.py install. Is this something you see as well?

File
"c:\python\ipydevel\venv\py32\lib\site-packages\ipython-0.13.dev-py3.2.egg\share\doc\ipython\examples\parallel\wa
ve2D\RectPartitioner.py", line 49
print 'The input global_num_cells is not ok!'
^
SyntaxError: invalid syntax

[takluyver]

Yes, I keep meaning to go and work out what's causing them - we solved it before, but when we unified setup.py, they came back. The installation seems to succeed despite the syntax errors, which is why I've not got round to looking at them yet.

[takluyver]

Thanks for getting this set up, by the way. I'll e-mail you the URL you need to trigger ShiningPanda (it contains a 'secret' token).

[jstenar]

When I run the testsuite with the --with-xml-coverage option I get one more error than when I run it without. See output from test run below, it is the "Test safe_execfile with non-ascii path" that I don't get when running without the coverage option:

IPython test group: IPython.core
.........................................K..............................................................................
....................................................................E................S.............F....................
...................................................................................................................SS...
.......KS..S....
======================================================================
ERROR: Test safe_execfile with non-ascii path
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\tests\test_interactiveshe
ll.py", line 264, in test_1
    _ip.shell.safe_execfile(self.fname, {}, raise_exceptions=True)
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\interactiveshell.py", lin
e 2333, in safe_execfile
    py3compat.execfile(fname,*where)
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\utils\py3compat.py", line 168,
 in execfile
    exec compile(scripttext, filename, 'exec') in glob, loc
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\coverage\control.py", line 249, in _should_trace
    canonical = self.file_locator.canonical_filename(filename)
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\coverage\files.py", line 48, in canonical_filename
    g = os.path.join(path, f)
  File "C:\python\ipydevel\VENV\Py27\lib\ntpath.py", line 108, in join
    path += "\\" + b
UnicodeDecodeError: 'ascii' codec can't decode byte 0x86 in position 1: ordinal not in range(128)

======================================================================
FAIL: Test that prun does not clobber string escapes (GH #1302)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\nose\case.py", line 197, in runTest
    self.test(*self.arg)
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\testing\decorators.py", line 2
28, in skipper_func
    return f(*args, **kwargs)
  File "C:\python\ipydevel\VENV\Py27\lib\site-packages\ipython-0.13.dev-py2.7.egg\IPython\core\tests\test_magic.py", lin
e 396, in test_prun_quotes
    nt.assert_equal(_ip.user_ns['x'], '\t')
AssertionError: ' ' != '\t'
    "' ' != '\\t'" = '%s != %s' % (safe_repr(' '), safe_repr('\t'))
    "' ' != '\\t'" = self._formatMessage("' ' != '\\t'", "' ' != '\\t'")
>>  raise self.failureException("' ' != '\\t'")


----------------------------------------------------------------------
Ran 380 tests in 4.498s

FAILED (KNOWNFAIL=2, SKIP=5, errors=1, failures=1)
**********************************************************************

[jstenar]

Shouldn't these two ways of doing prun give the same result?

In [4]: ip.magic("prun -q x = '\t'")

In [5]: x
Out[5]: ' '

In [6]: prun -q x = '\t'

In [7]: x
Out[7]: '\t'

[takluyver]

I'm not sure without looking in to how we handle escaping. Let's get the tests uploading to ShiningPanda, then make sure we've got issues open for anything that's failing.

[jstenar]

I believe the error when using --with-xml-coverage is caused by inserting a unicode string in sys.path somewhere in the test. I'm not sure if we should consider this a bug in our code or in coverage.py

[jstenar]

This is now our only failing test on windows. I'm trying to understand what the test is doing.

This is the line I'm tryin to understand:

_ip.magic("prun -q x = '\t'")

I interpret it as being equivalent to a %prun command at the command line where there is an actual tab-character inside the single quotes. Is that something we want to test for? Shouldn't that line be:

_ip.magic(r"prun -q x = '\t'")

At least that fixes the error for me on windows.

[fperez]

I think you're right, @jstenar... Go ahead with that change and we'll run test_pr quickly on it to check all is OK on linux with python2/3. If so, we can then get back to a fully passing Windows test suite, that would make me very happy.

[jstenar]

Done in PR #1913