Error when get_targets returns a large dict

[jaraco]

Originally reported by: Ben Sully (Bitbucket: sd2k, GitHub: sd2k)

---

The following error occurred when I had a get_targets function returning a large dict (~663 keys):

#!python

12:08:00 publisher.1    | Traceback (most recent call last):
12:08:00 publisher.1    |   File "/home/vagrant/.virtualenvs/mettle/bin/mettle", line 9, in <module>
12:08:00 publisher.1    |     load_entry_point('mettle==0.6.0', 'console_scripts', 'mettle')()
12:08:00 publisher.1    |   File "/vagrant/Dropbox/cygwin64/home/Ben/repos/work/mettle-server/mettle/cli.py", line 34, in main
12:08:00 publisher.1    |     args.command()
12:08:00 publisher.1    |   File "/vagrant/Dropbox/cygwin64/home/Ben/repos/work/mettle-server/mettle/cli.py", line 84, in run_publisher
12:08:00 publisher.1    |     publisher.main()
12:08:00 publisher.1    |   File "/vagrant/Dropbox/cygwin64/home/Ben/repos/work/mettle-server/mettle/publisher.py", line 57, in main
12:08:00 publisher.1    |     payload = get_record_as_json(curs, table, payload['id'])
12:08:00 publisher.1    |   File "/vagrant/Dropbox/cygwin64/home/Ben/repos/work/mettle-server/mettle/publisher.py", line 121, in get_record_as_json
12:08:00 publisher.1    |     cur.execute(q, (row_id,))
12:08:00 publisher.1    | psycopg2.ProgrammingError: missing FROM-clause entry for table "pipeline_runs"
12:08:00 publisher.1    | LINE 3:     FROM (SELECT pipeline_runs.*, 'pipeline_runs' as tablena...

The error seems to be in the query in get_record_as_json of publisher.py.

---

• Bitbucket: https://bitbucket.org/yougov/mettle/issue/1

[jaraco]

Original comment by Brent Tubbs (Bitbucket: btubbs, GitHub: btubbs):

---

I've just committed a fix for the SQL syntax bug reported here, but really long payloads seem to be causing problems with RabbitMQ. We might need another approach for dealing with lots of data in target_parameters.

05:15:53 publisher.1     | Traceback (most recent call last):
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/bin/mettle", line 9, in <module>
05:15:53 publisher.1     |     load_entry_point('mettle==0.6.3', 'console_scripts', 'mettle')()
05:15:53 publisher.1     |   File "/vagrant/Y/mettle-server/mettle/cli.py", line 30, in main
05:15:53 publisher.1     |     args.command()
05:15:53 publisher.1     |   File "/vagrant/Y/mettle-server/mettle/cli.py", line 58, in run_publisher
05:15:53 publisher.1     |     publisher.main()
05:15:53 publisher.1     |   File "/vagrant/Y/mettle-server/mettle/publisher.py", line 69, in main
05:15:53 publisher.1     |     delivery_mode=PIKA_PERSISTENT_MODE,
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 572, in basic_publish
05:15:53 publisher.1     |     (properties, body), False)
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 1159, in _send_method
05:15:53 publisher.1     |     self.connection.send_method(self.channel_number, method_frame, content)
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 274, in send_method
05:15:53 publisher.1     |     self._send_method(channel_number, method_frame, content)
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/connection.py", line 1503, in _send_method
05:15:53 publisher.1     |     self._send_frame(frame.Method(channel_number, method_frame))
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/adapters/blocking_connection.py", line 417, in _send_frame
05:15:53 publisher.1     |     super(BlockingConnection, self)._send_frame(frame_value)
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/connection.py", line 1486, in _send_frame
05:15:53 publisher.1     |     marshaled_frame = frame_value.marshal()
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/frame.py", line 74, in marshal
05:15:53 publisher.1     |     pieces = self.method.encode()
05:15:53 publisher.1     |   File "/home/vagrant/.virtualenvs/mettle/local/lib/python2.7/site-packages/pika/spec.py", line 1839, in encode
05:15:53 publisher.1     |     pieces.append(struct.pack('B', len(value)))
05:15:53 publisher.1     | struct.error: ubyte format requires 0 <= number <= 255
05:15:53 publisher.1     | exited with code 1

[jaraco]

Original comment by Brent Tubbs (Bitbucket: btubbs, GitHub: btubbs):

---

Actually, my test there was bad. I was using a service with a really long name to test large payloads, but that meant I also ended up with a really big routing key, which threw that Pika error.