Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

start with csvs and write hdf5 outputs if desired #290

Merged
merged 10 commits into from Feb 3, 2020
Merged

start with csvs and write hdf5 outputs if desired #290

merged 10 commits into from Feb 3, 2020

Conversation

bstabler
Copy link
Contributor

@bstabler bstabler commented Dec 31, 2019 •
edited

#248, #271, #278. We also updated the example to start with CSVs.

@coveralls
Copy link

Coverage Status

Coverage increased (+0.1%) to 86.392% when pulling 6336839 on RSGInc:develop into efc2bc6 on ActivitySim:develop.

@bstabler bstabler changed the title I/O improvements start with csvs and write hdf5 outputs if desired Feb 3, 2020
@bstabler bstabler merged commit 32dc999 into ActivitySim:develop Feb 3, 2020
bstabler added a commit that referenced this pull request Feb 4, 2020
@bstabler
more flexible time periods, mandatory schedule specs, input and outpu…
a3dca1f 
…t formats (#295)

* Provide more flexibility for defining mandatory schedule specifications.

* provide more flexibility for time periods.

* Improve consistency in slicing and fix code syntax issues.

* Consistency and code syntax.

* code syntax.

* code syntax fixes.

* syntax fix.

* Add necessary change to test script settings.

* Modify relevant section of the settings.yml for tests and examples.

* restore 'hours' as deprecated for backwards compatibility.

* Add default value for period_minutes for backwards compatibilitiy.

* create develop branch so we can merge features here before merging (releasing) to master

* provide more flexibility for time periods.

* Improve consistency in slicing and fix code syntax issues.

* Consistency and code syntax.

* code syntax fixes.

* Modify relevant section of the settings.yml for tests and examples.

* restore 'hours' as deprecated for backwards compatibility.

* Add default value for period_minutes for backwards compatibilitiy.

* Fix overflows

* Complete testing of the flexible time periods.

* pep8

* test some floats too.

* pep8

* start with csvs and write hdf5 outputs if desired  (#290)

* add h5_store option to write_tables

* keep csv files as the default

* read input tables from CSV

* update docs, drop scripts no longer needed, include example files in package

* formatting

Co-authored-by: Blake <brosenthalpdx@gmail.com>

* Update travis.yml

* Pandas 1.0 fixes

Co-authored-by: Clint Daniels <clint.daniels@wsp.com>
Co-authored-by: Blake <brosenthalpdx@gmail.com>
bstabler added a commit that referenced this pull request Jan 12, 2021
@bstabler @toliwaga @jpn--
Develop (#366)
7af0868 
* Provide more flexibility for defining mandatory schedule specifications.

* provide more flexibility for time periods.

* Improve consistency in slicing and fix code syntax issues.

* Consistency and code syntax.

* code syntax.

* code syntax fixes.

* syntax fix.

* Add necessary change to test script settings.

* Modify relevant section of the settings.yml for tests and examples.

* restore 'hours' as deprecated for backwards compatibility.

* Add default value for period_minutes for backwards compatibilitiy.

* create develop branch so we can merge features here before merging (releasing) to master

* provide more flexibility for time periods.

* Improve consistency in slicing and fix code syntax issues.

* Consistency and code syntax.

* code syntax fixes.

* Modify relevant section of the settings.yml for tests and examples.

* restore 'hours' as deprecated for backwards compatibility.

* Add default value for period_minutes for backwards compatibilitiy.

* Fix overflows

* Complete testing of the flexible time periods.

* pep8

* test some floats too.

* pep8

* start with csvs and write hdf5 outputs if desired  (#290)

* add h5_store option to write_tables

* keep csv files as the default

* read input tables from CSV

* update docs, drop scripts no longer needed, include example files in package

* formatting

Co-authored-by: Blake <brosenthalpdx@gmail.com>

* Update travis.yml

* Pandas 1.0 fixes

* Location and mode choice logsums (#298)

* optional logsums for location_choice and tour destination models

* mode choice logsums for trous and trips

* trip DEST_CHOICE_SAMPLE_TABLE_NAME

* update docs for writing logsums and also a couple small development page updates

* fix small py2 py 3 compatibility issue

* py2 py3 compatibility

Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>

* Deprecate Py2 and add CLI

* Tncs (#14)

* add tncs

* lognormal_for_df broadcasts duplicate indexes

* lognormal_for_df broadcasts duplicate indexes code

* correct lognormal calculations and add functions to calculation lognormal location and scale

* clean-up tnc wait time expressions

* set taxi, tnc wait time standard deviations to zero until we have better data

* add scaled option to random.lognormal_for_df

* add mobility-as-a-service modes to docs

* updated tnc mode constants

* rename maas to ridehail

Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>

* write trip matrices (#311)

require newer pandas

add ability to save trips table to pipeline and begin to finish example

all time periods

fix trip scheduling bug

add support for odt skims as well

Co-authored-by: Ben Stabler <benstabler@yahoo.com>

Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>

* small fixes for semcog deployment (#319)

* improved validation diagnostics in trip_purpose and various windows-related int32/int64 conversions

* pycodestyle

* deprecations in test code

* fix bug in handling of no viable trips case in choose_trip_destination

* add trace folder to example_mtc output folder

Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>

* major work on phase 5 (#325)

* estimation through atwork_subtour_mode_choice

* Tnc updates and notebooks (#18)

* move other resources into folder since examples now part of package as well

* add shapefile

* clean up folder setup

* estimation notebooks for larch (#19)

* multiprocessing related logging and error checking

* trip_destination handle all trips fail

* skim caching with numpy memmap to speed skim loading

* better chunking in vectorize_tour_scheduling

* Cli (#22)

* use activitysim_resources

Fixed auto sufficiency conditions in tour_mode_choice.csv. (Issue #324)

* increment to version 0.9.5

* correct write trip matrices sampling expansion and add vehicle occupancy to the expression file

* updates to documentation for phase 5 work except multiple zone systems

* additional updates for estimation integration  (#328)

* estimation integration updates #327
* drop module no longer needed

* update license, fix univ coeff bug, update estimation example (#339)

* improve example setup for specifying input tables (#27)
* improve LICENSE (#30)
* correct univ coeff template lookup (#28)

* add license to package (#341)

* move notebooks to examples folder so they are installed and update docs as a result (#343)

* correct doc links

* Address Minor Issues Associated with ARC Update to Develop Branch (#342)

* publish v0.9.5 (#340)

* update docs, drop scripts no longer needed, include example files in package

* Pandas 1.0 fixes

* Location and mode choice logsums (#298)

* Tncs (#14)

* write trip matrices (#311)

* small fixes for semcog deployment (#319)

* improved validation diagnostics in trip_purpose and various windows-related int32/int64 conversions

* fix bug in handling of no viable trips case in choose_trip_destination

* major work on phase 5 (#325)

* estimation through atwork_subtour_mode_choice

* Tnc updates and notebooks (#18)

* move other resources into folder since examples now part of package as well

* add example zone shapefile

* estimation notebooks for larch (#19)

* multiprocessing related logging and error checking

* trip_destination handle all trips fail

* skim caching with numpy memmap to speed skim loading

* better chunking in vectorize_tour_scheduling

* Cli (#22)

* use activitysim_resources

Fixed auto sufficiency conditions in tour_mode_choice.csv. (Issue #324)

* correct write trip matrices sampling expansion and add vehicle occupancy to the expression file

* additional updates for estimation integration  (#328)

* improve LICENSE (#30)

* correct univ coeff template lookup (#28)

Co-authored-by: Clint Daniels <clint.daniels@wsp.com>
Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>

* Remove duplicate coefficient.

* move coefficient processing up

* Update joint_tour_destination.py

* Fix reference to joint tour destination.yaml

* Error check for COEFFICIENT files

* Add person_id to joint choosers.

* move notebooks to examples folder so they are installed and update docs as a result (#343)

* publish (#344)

* move notebooks to examples folder so they are installed and update docs as a result (#343)

* correct doc links

* Remove duplicate coefficient.

* move coefficient processing up

* Update joint_tour_destination.py

* Fix reference to joint tour destination.yaml

* Error check for COEFFICIENT files

* Add person_id to joint choosers.

* Remove duplicate coefficient.

* move coefficient processing up

* Update joint_tour_destination.py

* Fix reference to joint tour destination.yaml

* Error check for COEFFICIENT files

* Add person_id to joint choosers.

Co-authored-by: Ben Stabler <bstabler@users.noreply.github.com>
Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>
Co-authored-by: bstabler <benstabler@yahoo.com>

* fix #288 chunk joint_touyr_participation by chunk_id and update tests to catch such errors

* gitignore trivia

* Support for Multiple Zone Systems and Transit Virtual Path Building (#362)

* TransitVirtualPathBuilder first cut - get_tvpb_logsum computes logsums on demand

* tour_mode_choice logsums for 3-zone working for school and workplace location

* z3 running all models, but with hacked trip_mode_choice (no coefficient templates)

* z3 running all models

* passing tests

* pcodestyle passes

* refactored test teardown

* tvpb tracing

* tvpb tracing roughout

* build multizone test data before running test

* read_settings_file support include_settings

* read_settings_file support include_settings

* move constants to configs yaml and expressions from abm.util to core

* tvpb estimate_chunk_overhead

* adaptive chunking

* fixed bug in trip scheduling when no trips in leg after iteration

* 3 zone tap_tap uniquify

* tvpb TableCache

* tvpb TableCache - all fiiels

* 3 zone with feather cache

* tidy up notes around creating the marin example

* memmap skims

* memmap skims

* correct drive transit cost expression for miles/feet.  Still need to add some additional missing expressions now that everything appears to be working correctly.

* skim docstrings

* disable 3 zone tap_tap caching for traced taps

* initialize_los

* 3 zone mp plumbing untested

* THREE_ZONE tvpb multiprocessing tests and fixes

* THREE_ZONE tvpb multiprocessing tests and fixes

* reanme core tvpb and cache modules

* reanme core tvpb and cache modules

* util.iprod alias for np.prod with dtype int64

* THREE_ZONE cache initialization bug bixes

* multizone bug fixes

* bugs

* multiprocessing mjultizone tweaks

* tvpb cacha\e as array

* complain if apporting with more process than slice rows

* complain if apporting with more process than slice rows

* chunk initialize_tvpb

* shorter trace file names because windows

* refactor pathbuilder sans DYNAMIC

* fix tour_scheduling_calc_row_size

* fix multiprocess resume across mp step bug

* pycodestyle

* minor chunk cleanup

* allow MemMapSkimFactory multiprocessing

* consolidate mtc and multizone configs

* commit updated marin example config files

* some more marin example config updates

* add validation summaries script

* some additional marin example updates

* update example skim time period settings to align with tm1 (#360)

* update example settings to align with tm1

* correct home_is_rural coding since tm1 was incorrect

* corrected skim time periods go in network_los.yaml now

* travis depends on these scripts for testing

* improve logging and increase travis timeout

* #359 (#364)

* updates for CDAP person type mapping

* allow duplicate time labels

* fix a copy small items for #359

* test updates

* pycodestyle

* 3 zone test fix

Co-authored-by: Clint Daniels <clint.daniels@wsp.com>

* longer travis timeout and avoid file permissions between tests related to caching

* test-travis

* ls fix

* add timeout

* fix path

* put in both locations for now

* reduce household sample sizes for tests to avoid travis timeout

* not sure what's taking so long with the second and third tests.  It runs in 40 minutes with py 3.7 on my computer as well

* linux file paths

* cli test depends on py 3.7+

* add coveralls back

* test new architecture

* try build matrix

* setup environments

* correct matrix expansion

* correct duplicate key

* correct key

* correct env usage

* no need to specify architecture

* update inject settings for tests

* updated examples to use activitysim resources repo

* rename folder

* update survey test files so all example_manifest.yaml examples now running

* updated notebooks, default settings for examples, and updated rst files

* fix a couple typos

* add comments to travis script

* ARC Updates (#365)

Add parking_location_choice, trip_departure_choice, and trip_scheduling_choice models.

* add example_mtc_arc_extensions so arc extensions can be run and tested

* small doc update

* fix arc extensions example

Co-authored-by: Clint Daniels <clint.daniels@wsp.com>
Co-authored-by: Blake <brosenthalpdx@gmail.com>
Co-authored-by: Blake Rosenthal <blake.rosenthal@rsginc.com>
Co-authored-by: Jeff Doyle <toliwaga@gmail.com>
Co-authored-by: Jeffrey Doyle <jeff.doyle@rsginc.com>
Co-authored-by: Jeffrey Newman <jeff@newman.me>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@bstabler @coveralls