Load up npm #4153

chadwhitacre · 2016-10-22T00:04:33Z

Part of #4148.

We need to load npm into our database and keep it up to date, in order to enable the /on/npm/, /on/npm/foo/, and /search changes we want to make.

aandis · 2016-10-22T15:16:49Z

This needs to have the work done so far in #4155 so I'm just gonna grab it from there.

chadwhitacre · 2016-10-22T15:20:42Z

@aandis Yeah, and we can drop the /on/npm/ stuff from this. Probably best to rebase here so we have a cleaner history. Are you comfortable with rebasing? :-)

aandis · 2016-10-22T15:37:56Z

Rebased to remove /on/npm work. Previous head was 4d92a22

aandis · 2016-10-22T15:57:43Z

Rebased again on top of #4155 to bring in all work from there. Commits show on https://github.com/gratipay/gratipay.com/pull/4153/commits but not here. weird.

aandis · 2016-10-22T16:06:47Z

Actually let's scratch that rebase. I wanted to use Package.insert but we'll need a multi insert statement here.

aandis · 2016-10-22T16:16:07Z

@whit537 how do I run the chomp script?

chadwhitacre · 2016-10-22T16:25:58Z

how do I run the chomp script?

@aandis Once you pip install -e . (as under make env) you should have it on your virtualenv's PATH via an entry_point in setup.py. We should write tests for all the pieces that go into it, though. I tried to factor it so it's easy to swap out the HTTP request for all to test with a much smaller subset of the full JSON.

aandis · 2016-10-22T16:26:22Z

Got it. env/bin/chomp

chadwhitacre · 2016-10-22T16:29:48Z

I wanted to use Package.insert but we'll need a multi insert statement here.

We may need to think differently about initial load and subsequent updates. For initial load we'll have a lot to do and it might be worth looking at COPY. Remember that for subsequent runs we'll need to update as well as insert. Sounds like we may not need to worry about delete (once a name is claimed, even if unpublished it gets a security package).

chadwhitacre · 2016-10-22T16:34:45Z

I had figured we could pass -1 as the arg to chomp to mean "reprocess everything".

Fetching and processing readmes is going to add significant expense. It may make sense to decouple that and just load the database with all and then process readmes.

aandis · 2016-10-22T16:36:49Z

it might be worth looking at COPY

I thought COPY could only be used with files. looking into it.

chadwhitacre · 2016-10-22T16:47:29Z

So maybe we have these three?

A script that takes all and spits out whatever COPY wants. We run that manually to initially populate the table (will we be able to use that same COPY method when adding packages from other package managers like RubyGems and PyPI? Or does that only work for populating new/empty tables?).
A script that takes all and upserts any record with modified time newer than the last time the script was run (every 30 minutes?).
A script that does select * from packages where readme is null order by mtime asc limit 1 and then hits the registry api and processes with markdown.marky. It runs continuously (either via cron, or with its own eternal loop).

I was thinking (1) and (2) could be variants of the same chomp script. Probably (3) is a separate script?

aandis · 2016-10-22T17:00:13Z

A script that takes all and spits out whatever COPY wants. We run that manually to initially populate the table

Manual work shouldn't be needed.

(will we be able to use that same COPY method when adding packages from other package managers like RubyGems and PyPI? Or does that only work for populating new/empty tables?).

COPY FROM copies data from a file to a table (appending the data to whatever is in the table already)

https://www.postgresql.org/docs/current/static/sql-copy.html

chadwhitacre · 2016-10-22T17:01:57Z

Manual work shouldn't be needed.

Okay. I mostly meant that we can just run that script as a one-off rather than having to install it in a long-lived dyno/cronjob/whatever.

aandis · 2016-10-22T17:01:58Z

A script that takes all and upserts any record with modified time newer than the last time the script was run (every 30 minutes?).

Do we get modified_time for a package inside all?

chadwhitacre · 2016-10-22T17:03:04Z

Do we get modified_time for a package inside all?

Yes!

aandis · 2016-10-22T17:12:50Z

Sounds good. I am gonna start working on 1. Might doze off in an hour. Will pick up tomorrow from where you leave it.

chadwhitacre · 2016-10-22T17:14:24Z

Awesome, @aandis, sounds good. I have #4117 just about topped up. I'll jump in here in a bit ...

aandis · 2016-10-22T18:31:40Z

@whit537 built up a bit on 1. Should need to call copy with the data created to insert it. Gonna head to bed now. Will pick it up tomorrow. :)

chadwhitacre · 2016-10-24T03:27:52Z

sql/branch.sql

+    ( id            bigserial   PRIMARY KEY
+    , package_id    bigint      NOT NULL REFERENCES packages ON UPDATE CASCADE ON DELETE RESTRICT
+    , email         text        NOT NULL
+    , UNIQUE (package_id, email)


I don't think we want a UNIQUE here because a package can have multiple author/maintainer emails, no?

Wouldn't the emails be different? This is just a way of preventing duplicate inserts of a email for a given package.

Yeah, you're right. I wasn't thinking straight. :)

chadwhitacre · 2016-10-24T03:28:18Z

sql/branch.sql

+
+    CREATE TABLE packages
+    ( id                    bigserial                   PRIMARY KEY
+    , package_manager_id    bigint                      NOT NULL REFERENCES package_managers ON UPDATE CASCADE ON DELETE RESTRICT


Pretty sure we don't need the ON UPDATE ... ON DELETE since we're referencing an immutable field.

chadwhitacre · 2016-10-24T03:28:41Z

gratipay/utils/markdown.py


 def render(markdown):
    return Markup(m.html(
        markdown,
        extensions=m.EXT_AUTOLINK | m.EXT_STRIKETHROUGH | m.EXT_NO_INTRA_EMPHASIS,
        render_flags=m.HTML_SKIP_HTML | m.HTML_TOC | m.HTML_SMARTYPANTS | m.HTML_SAFELINK
    ))
+
+def marky(markdown):
+    return subprocess.call()


This should get rebased on master now that #4154 is in.

chadwhitacre · 2016-10-24T11:25:40Z

I'm gonna dive into this today on the plane.

chadwhitacre · 2016-10-24T15:50:29Z

Here's a few commits. More later ...

chadwhitacre · 2016-10-26T13:00:14Z

Just checking ...

chadwhitacre · 2016-10-26T14:07:36Z

Yes! 💃

~ $ curl $URL | sync-npm serialize /dev/stdin | sync-npm upsert /dev/stdin
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   279  100   279    0     0   6578      0 --:--:-- --:--:-- --:--:--  6642
skipping None
skipping {u'description': '', u'package_manager': 'npm', u'emails': [], u'name': '_updated'}
processed 1 packages in   0 seconds
~ $

chadwhitacre · 2016-10-26T14:09:07Z

Double yes! 💃 💃

tranquil-ocean-34844::DATABASE=> select * from packages;
┌────┬─────────────────┬─────────────────┬───────────────────────┬────────┬────────────┬─────────────┬────────
│ id │ package_manager │      name       │      description      │ readme │ readme_raw │ readme_type │        
├────┼─────────────────┼─────────────────┼───────────────────────┼────────┼────────────┼─────────────┼────────
│  2 │ npm             │ testing-package │ A package for testing │        │            │             │ {alice@
└────┴─────────────────┴─────────────────┴───────────────────────┴────────┴────────────┴─────────────┴────────
(1 row)

lines 1-7/7 (END)

chadwhitacre · 2016-10-26T14:09:28Z

Okay! Let's light this candle! 🕯️

This location is already in LD_LIBRARY_PATH.

chadwhitacre · 2016-10-26T14:20:19Z

I've set up a 10-minute scheduler job on the test app. If that works then my plan is to:

switch to the real URL
merge and deploy this PR
schedule a daily run for now

chadwhitacre · 2016-10-26T14:20:35Z

YEEEEEEEEEEEEEEEEEEE

chadwhitacre · 2016-10-26T14:27:45Z

I'm going to land this as-is, and reticket loading up readmes as well as incrementalizing updates.

chadwhitacre · 2016-10-26T14:30:57Z

Ready to go once Travis is green ...

chadwhitacre · 2016-10-26T14:33:02Z

Okay! Here we go! ...

chadwhitacre · 2016-10-26T14:33:15Z

!m @aandis

!m *

chadwhitacre · 2016-10-26T15:31:11Z

Coming back from #4148 (comment)

processed 371932 packages in 308 seconds
Traceback (most recent call last):
  File "/app/.heroku/python/bin/sync-npm", line 9, in <module>
    load_entry_point('gratipay', 'console_scripts', 'sync-npm')()
  File "/app/gratipay/package_managers/sync.py", line 137, in main
    globals()[args.command](args)
  File "/app/gratipay/package_managers/sync.py", line 101, in upsert
    'FROM STDIN WITH (FORMAT csv)', fp)
psycopg2.IntegrityError: null value in column "description" violates not-null constraint
DETAIL:  Failing row contains (2, npm, 0, null, , , , {zolikahan@gmail.com}).
CONTEXT:  COPY updates, line 1: "npm,0,,"{""zolikahan@gmail.com""}""

chadwhitacre · 2016-10-26T15:31:17Z

More later!

chadwhitacre · 2016-10-26T21:44:54Z

Local. 👍

gratipay-test=# select count(*) from packages;
┌────────┐
│ count  │
├────────┤
│ 372147 │
└────────┘
(1 row)

chadwhitacre · 2016-10-26T23:52:12Z

Okay! Now what's up with the tests?

chadwhitacre · 2016-10-26T23:52:32Z

Picking up in #4158 ...

This was referenced Oct 22, 2016

Stub out Package model #4155

Merged

Integrate npm #4148

Closed

Wrap marky-markdown #4154

Merged

aandis force-pushed the chomp branch from 4d92a22 to 0640d05 Compare October 22, 2016 15:36

aandis force-pushed the chomp branch from 0640d05 to 49dca3b Compare October 22, 2016 15:51

aandis force-pushed the chomp branch from 49dca3b to 0640d05 Compare October 22, 2016 16:08

chadwhitacre added this to the Refocus on Corporate Sponsorship of Open Source milestone Oct 22, 2016

aandis force-pushed the chomp branch from 0b5b2ae to bea8850 Compare October 23, 2016 17:14

chadwhitacre commented Oct 24, 2016

View reviewed changes

chadwhitacre added 8 commits October 26, 2016 09:06

Okay! Let the Heroku bashing commence!

2ec25ae

Groan. YAJL requires cmake. O.o

de7821d

We can only write to /app

d46ec97

Gosh. Big tarball to download 24 times a day! O.o

468e395

Curl writes to stdout

4077a38

Fiddle with checksum check

7f519c1

Fix sha256sum call

8cc5602

We don't have to test, it just fails

f4f1655

chadwhitacre added 2 commits October 26, 2016 10:09

Put libyajl where we can find it

03142c0

This location is already in LD_LIBRARY_PATH.

Add some comments

56ee188

Switch to the real URL! Ready to deploy! :O

cc8ea2c

chadwhitacre merged commit c6a3bab into master Oct 26, 2016

chadwhitacre deleted the chomp branch October 26, 2016 14:33

chadwhitacre mentioned this pull request Oct 26, 2016

Core Radar 82 gratipay/inside.gratipay.com#863

Closed

chadwhitacre added the Review label Oct 27, 2016

Load up npm #4153

Load up npm #4153

Conversation

chadwhitacre commented Oct 22, 2016 • edited

aandis commented Oct 22, 2016 • edited

chadwhitacre commented Oct 22, 2016

aandis commented Oct 22, 2016

aandis commented Oct 22, 2016

aandis commented Oct 22, 2016

aandis commented Oct 22, 2016

chadwhitacre commented Oct 22, 2016 • edited

aandis commented Oct 22, 2016 • edited

chadwhitacre commented Oct 22, 2016

chadwhitacre commented Oct 22, 2016

aandis commented Oct 22, 2016

chadwhitacre commented Oct 22, 2016 • edited

aandis commented Oct 22, 2016

chadwhitacre commented Oct 22, 2016

aandis commented Oct 22, 2016

chadwhitacre commented Oct 22, 2016

aandis commented Oct 22, 2016

chadwhitacre commented Oct 22, 2016

aandis commented Oct 22, 2016

chadwhitacre Oct 24, 2016

Choose a reason for hiding this comment

aandis Oct 24, 2016

Choose a reason for hiding this comment

chadwhitacre Oct 24, 2016

Choose a reason for hiding this comment

chadwhitacre Oct 24, 2016

Choose a reason for hiding this comment

chadwhitacre Oct 24, 2016

Choose a reason for hiding this comment

chadwhitacre commented Oct 24, 2016 • edited

chadwhitacre commented Oct 24, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 26, 2016

chadwhitacre commented Oct 22, 2016 •

edited

aandis commented Oct 22, 2016 •

edited

chadwhitacre commented Oct 22, 2016 •

edited

aandis commented Oct 22, 2016 •

edited

chadwhitacre commented Oct 22, 2016 •

edited

chadwhitacre commented Oct 24, 2016 •

edited