Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

github caching = fail #3

Closed
johnclary opened this issue Sep 8, 2016 · 1 comment
Closed

github caching = fail #3

johnclary opened this issue Sep 8, 2016 · 1 comment

Comments

@johnclary
Copy link
Member

github cdn seems to cache files for 5-10 minutes. what this means is that, although data may be published at very small intervals (e.g. 1 min) to the open data portal, these only 1 in 10ish of these events will be logged on github, which means the 'last updated' date in our application will not be accurate if it is pointing to the github logs.

options:

  1. publish logs on socrata rather than github
  2. do a complete refresh of dataset on socrata instead of an upsert
  3. leave as is and understand that the 'last updated' date in our application is +/- 10 minutes
@johnclary
Copy link
Member Author

Logging on Socrata now:
https://data.austintexas.gov/dataset/Signal-Status-Publication-Log-BETA-/n5kp-f8k4

johnclary added a commit that referenced this issue Jul 20, 2017
@johnclary
# This is a combination of 62 commits.
a015c08 
# This is the 1st commit message:
init

This is the initial commit for transportation-data-publishing. Actually, it's a squash of about 70 commits that are now prehistory. To new beginnings...

# This is the commit message #2:

refactor for config life

# This is the commit message #3:

ignore some bs notes

# This is the commit message #4:

delete a line

# This is the commit message #5:

a happy village of python scripts living working loving dying

# This is the commit message #6:

ping and update, then move on

# This is the commit message #7:

write device data to json

# This is the commit message #8:

write to json

# This is the commit message #9:

better config and mind numbing

# This is the commit message #10:

tiny bugs

# This is the commit message #11:

disable pandas csv for now

# This is the commit message #12:

proper log path join

# This is the commit message #13:

restore csv writing with pandas

# This is the commit message #14:

hello worlds

# This is the commit message #15:

hello worlds

# This is the commit message #16:

progreso

# This is the commit message #17:

progreso

# This is the commit message #18:

progreso

# This is the commit message #19:

kits sync

# This is the commit message #20:

hardly working

# This is the commit message #21:

sync knack > kits cctv

# This is the commit message #22:

modules aren't subscriptable bro

# This is the commit message #23:

log dest

# This is the commit message #24:

new log dest

# This is the commit message #25:

whoops debugging

# This is the commit message #26:

signal status pusher for the modern world

# This is the commit message #27:

a tree-hugging logger

# This is the commit message #28:

teach a naive date about the world

# This is the commit message #29:

traffic count processing

# This is the commit message #30:

output directory

# This is the commit message #31:

a quote with a primary key

# This is the commit message #32:

replaced with knack_data_pub

# This is the commit message #33:

query anything!

# This is the commit message #34:

agol > CIFS translation

# This is the commit message #35:

we have a feed

# This is the commit message #36:

file and dataset replacers

# This is the commit message #37:

remove this file

# This is the commit message #38:

logging and email alerts

# This is the commit message #39:

email alerts and traceback logging

# This is the commit message #40:

email alerts and traceback logging

# This is the commit message #41:

fix response parsing keyerror

# This is the commit message #42:

rename signal eng area

# This is the commit message #43:

trace it back

# This is the commit message #44:

handle empty features pt. 1

# This is the commit message #45:

logging and email

# This is the commit message #46:

cleanup

# This is the commit message #47:

logging, etc. ready to deploy

# This is the commit message #48:

rename row_id

# This is the commit message #49:

utc date formatting

# This is the commit message #50:

update for VM deploy

# This is the commit message #51:

happy loaders

# This is the commit message #52:

wip: cleanup

# This is the commit message #53:

WIP: cifs cleanup

# This is the commit message #54:

WIP: in-memory file writing

# This is the commit message #55:

WIP: tabular upsert instead of replace

# This is the commit message #56:

WIP: strip whitespace from input text

# This is the commit message #57:

WIP: use replace instead of strip when parsing csv

# This is the commit message #58:

WIP: logging, email alerts, ready to deploy

# This is the commit message #59:

WIP: more date/time fields

# This is the commit message #60:

WIP: del whitespace

# This is the commit message #61:

Format datetime as ISO 8601 with local timestamp

Prior to this change the datetime fields were stored in UTC time.
With this change datetime fiels are formated according to ISO 8601, with
US/Central timezone explictly indicated in the field value. E.g. 2016-06-15T10:00:00-05:00. The datets also have a TIME field, which was and contiues to be stored as hours, minutes, and seconds, in local time. Although we do not explicity inidicate the the TIME field is in local time, it is obvious thanks to the corresponding datetime field which does indicate the time

# This is the commit message #62:

WIP: tabular upsert instead of replace
johnclary added a commit that referenced this issue Jan 10, 2018
@johnclary
Containerize Script Deployment (cleanup #3)
f126e2a
johnclary added a commit that referenced this issue Jan 10, 2018
@johnclary
Merge pull request #115 from cityofaustin/cleanup2
93ed8f5 
Containerize Script Deployment (cleanup #3)
JoeyL6 added a commit that referenced this issue Dec 17, 2018
@JoeyL6
# This is a combination of 3 commits.
52071ec 
# This is the 1st commit message:
This commits is to add new postgre sql database to knack connection scripts that transfer signal preventive maintenance work order from postgre sql database to knack signal preventive maintenance object.

# The commit message #2 will be skipped:

#	delete set_tracce in replace method

# The commit message #3 will be skipped:

#	squash multiple small bug fixes and WIPs together
JoeyL6 added a commit that referenced this issue Dec 17, 2018
@JoeyL6
# This is a combination of 3 commits.
56ae2b4 
# This is the 1st commit message:
This commits is to add new postgre sql database to knack connection scripts that transfer signal preventive maintenance work order from postgre sql database to knack signal preventive maintenance object.

# The commit message #2 will be skipped:

#	fixed complete_by field not updating bug

# The commit message #3 will be skipped:

#	This commits is to add new postgre sql database to knack connection scripts that transfer signal preventive maintenance work order from postgre sql database to knack signal preventive maintenance object.
JoeyL6 added a commit that referenced this issue Dec 17, 2018
@JoeyL6
# This is a combination of 11 commits.
1c18fd8 
# This is the 1st commit message:
This commits is to add new postgre sql database to knack connection scripts that transfer signal preventive maintenance work order from postgre sql database to knack signal preventive maintenance object.

# The commit message #2 will be skipped:

#	fixed complete_by field not updating bug

# The commit message #3 will be skipped:

#	This commits is to add new postgre sql database to knack connection scripts that transfer signal preventive maintenance work order from postgre sql database to knack signal preventive maintenance object.

# The commit message #4 will be skipped:

#	Merge branch 'master' into postgre-knack

# The commit message #5 will be skipped:

#	squash multiple small bug fixes and WIPs together

# The commit message #6 will be skipped:

#	fixed complete_by field not updating bug

# The commit message #7 will be skipped:

#	fix app_name not find bug in update_signals_modified_time

# The commit message #8 will be skipped:

#	yet another app_name not find bug

# The commit message #9 will be skipped:

#	fix return type bug
#
#	format cleaning

# The commit message #10 will be skipped:

#	fix return type bug

# The commit message #11 will be skipped:

#	format cleaning
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@johnclary