Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .github/workflows/dev.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,22 @@ on:
pull_request:

jobs:
prettier:
runs-on: ubuntu-latest

steps:
- name: Checkout
uses: actions/checkout@v2
with:
# Make sure the actual branch is checked out when running on pull requests
ref: ${{ github.head_ref }}

- name: Prettify code
uses: creyD/prettier_action@v3.3
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think apache repos are limited to official github actions or ones that are local to this repo (there were some security issues previously):.

Notice: December 27, 2020: We only allow Actions that are official "Made by GitHub" or local to the Apache org on GitHub, to address a potential security vulnerability. This is an incident-related policy change. We are researching the situation, and the policy may evolve based on what we learn.

https://infra.apache.org/github-actions-secrets.html

Given https://github.com/creyD/prettier_action appears to be MIT licensed, however, what about copying the code for that action into this repo?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point. do you think copying it to be something under apache org repo is desired?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good point. do you think copying it to be something under apache org repo is desired?

Yes, I think that would be preferable (the rationale being that then that someone can't change the contents of the action in the third party repository and affect the apache org via a github action)

with:
prettier_options: --write {ballista,datafusion,datafusion-examples,dev,docs,python}/**/*.md
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
lint:
name: Lint C++, Python, R, Rust, Docker, RAT
runs-on: ubuntu-latest
Expand Down
2 changes: 1 addition & 1 deletion datafusion-examples/examples/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,4 +25,4 @@ The examples `csv_sql.rs` and `parquet_sql.rs` demonstrate building a query plan

## Distributed

The `flight-client.rs` and `flight-server.rs` examples demonstrate how to run DataFusion as a standalone process and execute SQL queries from a client using the Flight protocol.
The `flight-client.rs` and `flight-server.rs` examples demonstrate how to run DataFusion as a standalone process and execute SQL queries from a client using the Flight protocol.
14 changes: 12 additions & 2 deletions dev/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,14 +33,16 @@ committer.

## How to merge a Pull request

Please don't merge PRs using the Github Web interface. Instead, set up
your git clone such as to have a remote named ``apache`` pointing to the
Please don't merge PRs using the Github Web interface. Instead, set up
your git clone such as to have a remote named `apache` pointing to the
official Arrow repository:

```
git remote add apache git@github.com:apache/arrow.git
```

and then run the following command:

```
./dev/merge_arrow_pr.sh
```
Expand All @@ -64,10 +66,13 @@ If these aren't supplied, the script will ask you the values of them.
Note that the directory name of your Arrow git clone must be called `arrow`.

example output:

```
Which pull request would you like to merge? (e.g. 34):
```

Type the pull request number (from https://github.com/apache/arrow/pulls) and hit enter.

```
=== Pull Request #X ===
title Blah Blah Blah
Expand All @@ -77,16 +82,20 @@ url https://api.github.com/repos/apache/arrow/pulls/X

Proceed with merging pull request #3? (y/n):
```

If this looks good, type y and hit enter.

```
From git-wip-us.apache.org:/repos/asf/arrow.git
* [new branch] master -> PR_TOOL_MERGE_PR_3_MASTER
Switched to branch 'PR_TOOL_MERGE_PR_3_MASTER'

Merge complete (local ref PR_TOOL_MERGE_PR_3_MASTER). Push to apache? (y/n):
```

A local branch with the merge has been created.
type y and hit enter to push it to apache master

```
Counting objects: 67, done.
Delta compression using up to 4 threads.
Expand Down Expand Up @@ -115,6 +124,7 @@ Merge hash: 485658a5

Would you like to pick 485658a5 into another branch? (y/n):
```

For now just say n as we have 1 branch

## Verifying Release Candidates
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,9 @@ Revision: {revision}

Submitted crossbow builds: [{repo} @ {branch}](https://github.com/{repo}/branches/all?query={branch})

|Task|Status|
|----|------|
|docker-cpp-cmake32|[![CircleCI](https://img.shields.io/circleci/build/github/{repo}/{branch}-circle-docker-cpp-cmake32.svg)](https://circleci.com/gh/{repo}/tree/{branch}-circle-docker-cpp-cmake32)|
|wheel-osx-cp36m|[![TravisCI](https://img.shields.io/travis/{repo}/{branch}-travis-wheel-osx-cp36m.svg)](https://travis-ci.com/{repo}/branches)|
|wheel-osx-cp37m|[![TravisCI](https://img.shields.io/travis/{repo}/{branch}-travis-wheel-osx-cp37m.svg)](https://travis-ci.com/{repo}/branches)|
|wheel-win-cp36m|[![Appveyor](https://img.shields.io/appveyor/ci/{repo}/{branch}-appveyor-wheel-win-cp36m.svg)](https://ci.appveyor.com/project/{repo}/history)|
| Task | Status |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| docker-cpp-cmake32 | [![CircleCI](https://img.shields.io/circleci/build/github/{repo}/{branch}-circle-docker-cpp-cmake32.svg)](https://circleci.com/gh/{repo}/tree/{branch}-circle-docker-cpp-cmake32) |
| wheel-osx-cp36m | [![TravisCI](https://img.shields.io/travis/{repo}/{branch}-travis-wheel-osx-cp36m.svg)](https://travis-ci.com/{repo}/branches) |
| wheel-osx-cp37m | [![TravisCI](https://img.shields.io/travis/{repo}/{branch}-travis-wheel-osx-cp37m.svg)](https://travis-ci.com/{repo}/branches) |
| wheel-win-cp36m | [![Appveyor](https://img.shields.io/appveyor/ci/{repo}/{branch}-appveyor-wheel-win-cp36m.svg)](https://ci.appveyor.com/project/{repo}/history) |
9 changes: 5 additions & 4 deletions dev/benchmarking/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,6 @@
~ under the License.
-->


> NOTE: For those deploying this database, Postgres does not by default use
> UTF-8, however it is [required for the jsonb][pg-jsonb] format used in
> some columns to always work. This [stackoverflow post][so-utf8] describes
Expand Down Expand Up @@ -81,7 +80,6 @@ database will be set up automatically once the container is running.
To start the containers, be sure to have [Docker installed][docker],
and then run the following from this directory (arrow/dev/benchmarking).


```
docker-compose up
```
Expand Down Expand Up @@ -111,9 +109,11 @@ The `psql` shell client is bundled with the PostgreSQL core distribution
available from the [Postgres download page][postgres-downloads].
Using the `PG_USER` defined in the `.env` file (currently "benchmark"),
the command to connect to the container is:

```shell
psql -h localhost -p 5432 -U benchmark
```

There is an example script in [examples/example.sql](examples/example.sql) that
runs some queries against the database. To run it in the psql client, type
the following in the psql command-line interface:
Expand Down Expand Up @@ -168,14 +168,15 @@ The script [graphql_submit.sh](./graphql_submit.sh) simplifies submission
to the database via curl. Examples:

```shell
./graphql_submit.sh benchmarks examples/benchmark_example.json
./graphql_submit.sh benchmarks examples/benchmark_example.json
./graphql_submit.sh runs examples/benchmark_run_example.json
```

#### Querying

The output of the query is a JSON object that is hard to read on the command line.
Here is an example query in the shell:

```shell
curl -X POST \
-H "Content-Type: application/json" \
Expand All @@ -190,6 +191,7 @@ which (if you have previously run the "examples.sql" command) yields
```

Here is an example query using Python:

```python
import json
import requests
Expand Down Expand Up @@ -219,7 +221,6 @@ for row in response.json()['data']['allEnvironmentViews']['edges']:
> how to do it for Amazon RDS. This [section of the docs][pg-charset]
> states how to do it in general, i.e.: `initdb -E UTF8`.


## Quick reference

- String variables `'have single quotes'`
Expand Down
10 changes: 5 additions & 5 deletions dev/release/VERIFY.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,11 +34,11 @@ GLib, Java and JavaScript builds on Linux and macOS. Read the comments in

You need the followings to verify C GLib build:

* GLib
* GObject Introspection
* Ruby (not EOL-ed version is required)
* gobject-introspection gem
* test-unit gem
- GLib
- GObject Introspection
- Ruby (not EOL-ed version is required)
- gobject-introspection gem
- test-unit gem

You can install them by the followings on Debian GNU/Linux and Ubuntu:

Expand Down
2 changes: 0 additions & 2 deletions dev/tasks/conda-recipes/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -54,14 +54,12 @@ related parts (the cloning of arrow and the jinja templated variables) and
moving the matrix definitions like [this][matrix-definition] to the crossbow
[tasks.yml][../tasks.yml] config file.


### Porting recipes from crossbow to the upstream feedstocks

Theoretically these recipes should be up to date with the actual version of
Arrow, so during the release procedure the content of these recipes should be
copied to the upstream feedstocks.


[arrow-cpp-feedstock]: https://github.com/conda-forge/arrow-cpp-feedstock
[parquet-cpp-feedstock]: https://github.com/conda-forge/parquet-cpp-feedstock
[matrix-definition]: https://github.com/conda-forge/arrow-cpp-feedstock/blob/master/.azure-pipelines/azure-pipelines-linux.yml#L12
2 changes: 1 addition & 1 deletion dev/tasks/gandiva-jars/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,4 +26,4 @@ Do the following to update arrow manylinux docker image for building Gandiva Jar
- Export JAVA_HOME environment variable.
- Then update build_boost.sh under python/manylinux1/scripts to build boost statically.

Please look at https://github.com/praveenbingo/arrow/tree/buildGandivaDocker that already has these changes.
Please look at https://github.com/praveenbingo/arrow/tree/buildGandivaDocker that already has these changes.
6 changes: 3 additions & 3 deletions dev/tasks/linux-packages/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,9 @@

## Requirements

* Ruby
* Docker
* Tools to build tar.gz for Apache Arrow C++ and GLib
- Ruby
- Docker
- Tools to build tar.gz for Apache Arrow C++ and GLib

## How to build .deb packages

Expand Down