[Publish] processing speed

[anuveyatsu]

After pushing a dataset I have to wait for some time before my data gets processed on datahub, e.g.:

100kb:

needed 36 sec after link displayed
needed 35 sec after link displayed(person 2)

1mb:

needed 45 sec after link displayed
needed 42 sec after link displayed(person 2)

150mb:

needed 40 hours after link displayed

How to reproduce

• take this 100kb data https://github.com/frictionlessdata/test-data/blob/master/files/csv/100kb.csv
  • and push it
  • see how long it takes
• take this 1mb data https://github.com/frictionlessdata/test-data/blob/master/files/csv/1mb.csv
  • and push it
  • see how long it takes
• run script python csv-generator.py from here https://github.com/frictionlessdata/test-data
  • and push

Expected behavior

• processing small dataset 100Kb | processing time < 10 sec
• processing medium dataset 1Mb | processing time < 30 sec
• processing big dataset 1Gb | processing time < 10 min

[AcckiyGerman]

Renamed title from "push speed" to "processing speed" - the push speed itself is fast enough (do 2.5 Mb/sec on my pc, which is my internet provider limit)

[zelima]

Think < 10 seconds is really unrealistic goal here. We need to rethink and define the realistic goals
Results after latest changes:

• 100KB - needs 40 seconds for full process
  • 5 seconds for push
  • 35 seconds from link displayed in CLI until page reload
• 1mb - needs 45 seconds totally
  • 8 seconds for push
  • 37 seconds from link displayed in CLI until page reload

[AcckiyGerman]

100KB and 1mb are x10 size difference, but only 35 vs 37 = 2 sec processing time difference.
I assume there is some time-consuming operation, that is not related to the data size.

[AcckiyGerman]

@zelima consider using pypy instead of python interpreter on the server - it could be twice faster!

[AcckiyGerman]

@akariv WDYT about pypy interpreter? ^^^

[anuveyatsu]

Closing as FIXED since we have improved our processing speed and nobody has reported any speed related issues.